What is DOcloud?

DOCloud is the short name of the IBM Decision Optimization on Cloud service. This service lets you solve CPLEX and OPL problems on the Cloud. You can access the interactive service called DropSolve or you can use use the API to integrate the service into your application.

CPLEX is a mathematical programming solver for linear programming, mixed integer programming, and quadratic programming. CPLEX is known as the leader and most efficient solver of its kind. It is a prescriptive analytics tool that helps you take better decision about your business. OPL is a modelling language that facilitates the use of CPLEX. These tools used to be available on-premise only, with a nice framework to build application called Decision Optimization Center.

The good news is that you can now access these tools on-line!

DropSolve is a simple web interface where you can literally drag and drop your model and data files, and they will be executed on IBM servers. If you want to build an application, you can use the API. The API is pretty simple: you create a job with the model and data, then you submit and monitor it, and finally you get the results once it is completed. There is a developer community where you can access to the API documentation, the examples and a forum.

To help you get started, I have created a YouTube playlist where you will find tutorials. The blog "IT Best Kept Secret Is Optimization" is also a must-read about optimization and DOcloud.

How to retrieve incrementally a time series using a REST API?

In this post, I would like to propose a REST API pattern to retrieve incrementally a time series.

Let's take the example of a system generating a time series such as price change, log events, machine status change etc. This system is able to send a sequence of data points using a messaging protocol, but as with many messaging protocols, the delivery and the order of delivery are not guaranteed. The goal is to propose a service that will record these messages and provide a REST API to incrementally access existing data points or new data points when they are available and in order.

First of all, we can easily record these messages in a data store, let's say using JSON format and in a MongoDB collection or in a CouchDB database. We can also notice that messages should be ordered and assigned a sequence number at the source. This way, it is possible to tell if 2 messages are contiguous or if there are one or more messages not yet received in between. So the first part of the service is about subscribing to the data stream, and storing the messages as they come.

Then, the second part of the service is to provide a REST API to access the time series (no Websockets or long polling for now). Well, the REST API can be very simple as this one:

   GET https://www.myservice.com/timeseries/TS?start=X

where TS is the name of the time series and X the sequence index at which we need to start returning the data points. The returned data could be of this form with X=10:

[ 
  { "index" : 10,
    "date" : 1423259728987,
    "value" : 10},
  { "index" : 11,
    "date" : 1423259730094,
    "value" : 42},
  ...
]

However, there are a couple of things to consider:

• The list of data points can be very large, and could cause out-of-memory crashes on the client or on the server depending on the implementation.
• Data points are received asynchronously, and there is no guarantee that the returned list will contain a continuous sequence.
• Some data point may be lost and will not be delivered, and again there is no guarantee that the returned list will contain a continuous sequence.
• Finally, the data stream may have ended, so that no more data points will be available and we need to indicate this.

With this in mind, I would like to propose the following design:

• There must be a server side limit on the number of data points returned at a time. This limit can be a default and potentially a lower limit could be specified by the client, but the important thing is to define a reasonable maximum limit to enforce. With this approach, the client can iterate over the data points, and specify the next start index as the last index + 1
• As some data points may be received out of order, gaps in the sequence can happen. In this case, the returned list should stop at the first gap detected. So if we recorded data points X, X+1, X+2, X+4, the call would return only X, X+1 and X+2.
• But some data point may be lost, and we need to set a maximum delay when we will consider that the data point will not be returned. If we continue the previous example, the next call will be with X=X+3. If the elapsed time between X+4 and the current time is over a maximum delay, we should assume X+3 lost, and we can return a fake data point with an attribute missing set to true, along with the rest of the sequence.
• Finally, if we know there is no more data points because the stream ended, we can indicate this by returning a fake data item with an attribute stop set to true,

In conclusion, the client can poll the server until it receives the stop flag. At each call it will receive no more than the maximum block size defined. It can specify the next call by adding 1 to the last index received. The API also guarantees that the data items are returned in order, and that if a data item is not available after some maximum delay, it will be returned and flagged as missing along with the rest of the data points. I believe this approach can be of a general interest.

How to configure WAS Liberty to use Apache Wink and Jackson?

As we have seen in the previous post, IBM WebSphere Liberty comes with JAX-RS and JSON support. In this post, I will show you how to explicitly use Apache Wink for the JAX-RS runtime and use Jackson as the JSON provider instead of the default providers. The updated code can be found on my GitHub repository.

The first step is to update the maven dependencies to add Wink and Jackson like this:

<dependency>
  <groupId>org.apache.wink</groupId>
  <artifactId>wink-server</artifactId>
  <version>1.4</version>
</dependency>

<dependency>
  <groupId>com.fasterxml.jackson.jaxrs</groupId>
  <artifactId>jackson-jaxrs-json-provider</artifactId>
  <version>2.4.3</version>
</dependency>

Then you need to declare the servlet in your web.xml file. Instead of using the the WAS Liberty JAX-RS servlet, you just need to indicate the class name of the Apache Wink servlet.

<servlet>
  <description>JAX-RS Tools Generated - Do not modify</description>
  <servlet-name>JAX-RS Servlet</servlet-name>
  <servlet-class>org.apache.wink.server.internal.servlet.RestServlet</servlet-class>
  <init-param>
    <param-name>javax.ws.rs.Application</param-name>
    <param-value>com.mycloudtips.swagger.MctApplication</param-value>
  </init-param>
  <load-on-startup>1</load-on-startup>
  <enabled>true</enabled>
  <async-supported>false</async-supported>
</servlet>
<servlet-mapping>
  <servlet-name>JAX-RS Servlet</servlet-name>
  <url-pattern>/jaxrs/*</url-pattern>
</servlet-mapping>

And in the application class (MctApplication class) you need to add the Jackson provider.

    @Override
    public Set<Class<?>> getClasses() {
 Set<Class<?>> classes = new HashSet<Class<?>>();

 classes.add(ApiDeclarationProvider.class);
 classes.add(ResourceListingProvider.class);
 classes.add(ApiListingResourceJSON.class);

 classes.add(JacksonJsonProvider.class);
  
 return classes;
    }

Finally, make sure you remove the feature jaxrs-1.1 from your server.xml and replace it by a simple servlet-3.0. That's it, esay peasy.

How to document your JAX-RS API using Swagger, WAS Liberty Profile and Bluemix?

Swagger has become the de facto standard for REST API documentation. It is also a pretty generic framework and developers need to know how to configure their specific environment. In this post, I will review the steps required to document a JAX-RS API developed with IBM WebSphere Application Server Liberty Profile. The complete example is available on my GitHub repository.

I will assume that you have created a Maven Dynamic Web project in Eclipse (project name and web context root are set to 'swagger-liberty'), and that you have defined a WAS Liberty server environment. Setting up your environment is outside the scope of this post, but you can find more information here.

In order to develop and document your JAX-RS API, you will need to follow these steps:

• Declare the required maven dependencies.
• Declare the JAX-RS and Swagger servlets.
• Declare the Swagger JAX-RS providers and your JAX-RS resources.
• Implement and document your APIs using Java annotations.
• Copy the Swagger UI web resource files.
• Activate the JAX-RS feature of Liberty.
• Test your server locally.

The first step is to add the maven dependencies to your maven project. You need to add the Swagger JAX-RS bridge, logging bridge and JEE 6 apis:

<dependency>
  <groupId>com.wordnik</groupId>
  <artifactId>swagger-jaxrs_2.10</artifactId>
  <version>1.3.10</version>
</dependency>
<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-jdk14</artifactId>
  <version>1.7.7</version>
</dependency>
<dependency>
  <groupId>javax</groupId>
  <artifactId>javaee-web-api</artifactId>
  <version>6.0</version>
  <scope>provided</scope>
</dependency>

Then you need to declare the servlets in your web.xml file. The first servlet is used to indicate the JAX-RS runtime where to find your JAX-RS application.

<servlet>
  <description>JAX-RS Tools Generated - Do not modify</description>
  <servlet-name>JAX-RS Servlet</servlet-name>
  <servlet-class>com.ibm.websphere.jaxrs.server.IBMRestServlet</servlet-class>
  <init-param>
    <param-name>javax.ws.rs.Application</param-name>
    <param-value>com.mycloudtips.swagger.MctApplication</param-value>
  </init-param>
  <load-on-startup>1</load-on-startup>
  <enabled>true</enabled>
  <async-supported>false</async-supported>
</servlet>
<servlet-mapping>
  <servlet-name>JAX-RS Servlet</servlet-name>
  <url-pattern>/jaxrs/*</url-pattern>
</servlet-mapping>

The second servlet is used to configure the Swagger runtime and indicate where to find the API meta-data (the base path, which is made of the web root context and the JAX-RS servlet mapping).

<servlet>
  <servlet-name>DefaultJaxrsConfig</servlet-name>
  <servlet-class>com.wordnik.swagger.jaxrs.config.DefaultJaxrsConfig</servlet-class>
  <init-param>
	<param-name>api.version</param-name>
	<param-value>1.0.0</param-value>
  </init-param>
  <init-param>
     <param-name>swagger.api.basepath</param-name>
     <param-value>/swagger-liberty/jaxrs</param-value>
   </init-param>
   <load-on-startup>2</load-on-startup>
</servlet>

The application class (MctApplication class) is the place where you need to declare the JAX-RS Swagger providers and your JAX-RS resource (MctResource class). Note that I usually declare the resources as singletons so that they are not created at each request.

    @Override
    public Set<Class<?>> getClasses() {
	Set<Class<?>> classes = new HashSet<Class<?>>();

	classes.add(ApiDeclarationProvider.class);
	classes.add(ResourceListingProvider.class);
	classes.add(ApiListingResourceJSON.class);
	return classes;
    }
    @Override
    public Set<Object> getSingletons() {
	Set<Object> singletons = new HashSet<Object>();
	singletons.add(new MctResource());
	return singletons;
    }

The resource class is the place where you can develop and document your APIs. You need to use the JAX-RS and Swagger annotations. Here is an example to declare a method returning a list of books:

@GET
@ApiOperation(value = "Returns the list of books from the library.", 
              response = MctBook.class, responseContainer = "List")
@ApiResponses(value = { @ApiResponse(code = 200, message = "OK"),
	@ApiResponse(code = 500, message = "Internal error") })
public Collection<MctBook> getBooks() {
  return library.values();
}

The server will include the Swagger UI and you need to copy the web resources (index.html, o2c.html, sagger-ui.js, sawgger-ui.min.js, lib, images and css files and directories). You can find these files in the Swagger UI JAX-RS sample or in my GitHub repository. You also need to adjust a path in the index.html file to point to your API:

   $(function () {
      window.swaggerUi = new SwaggerUi({
      url: "/swagger-liberty/jaxrs/api-docs",
      ...
    });

At this point, your project should compile fine and you should get ready to test. Before doing so, you need to activate the JAX-RS support in Liberty. Remember that Liberty is very flexible and let you decide what features will be loaded. To do so, you need to add the jaxrs-1.1 feature in the server.xml file.

   <featureManager>
    	<feature>jaxrs-1.1</feature>
        <feature>localConnector-1.0</feature>
    </featureManager>

Finally, you can add your application to your server runtime and start it. You should then be able to access the Swagger UI:

http://localhost:9080/swagger-liberty/

As an optional step, not covered in this post, you can easily deploy this server to IBM Bluemix.