Why is Cloud Foundry a revolution?

I have recently attended the Cloud Foundry Summit 2015 in Santa Clara, and I wanted to share the excitement around this growing technology. First, last year there were something like 1000 attendees in San Francisco, and this year it was about 1500. This quickly shows the growing interest. And most importantly, you can really see that Cloud Foundry (CF) is being adopted by many public PaaS providers (Pivotal, IBM, HP, SAP, EMC, Huawei) and also by companies setting up internal or industry specific PaaS (GE, Comcast, AllState, Verizon, Lookeed Martin). IBM mentioned that Bluemix, their PaaS based on Cloud Foundry, was getting 8000 new users per week. So what is going on and why is this wave coming? Here are several reasons showing the benefits of Cloud Foundry:

CF empowers developers

There is no question that CF empowers developers by letting them develop and deploy their ideas very quickly. No need to request machines, setup network and access rights etc. IBM bluemix is an example of that and once you've got your account set up you can deploy in minutes. Also, because it is so easy to deploy, it is a revolution for prototyping and testing. Instead of deploying to a shared test system, a developer can deploy all the components into his/her own space. To some extent, there is no more the need to mock dependent services, and you can test with the real components.

CF reduces barriers between Dev and Ops towards the DevOps model

In the end, the goal is to develop and deploy applications that are useful, easy to use and robust. The separation between Dev and Ops does not work anymore. We need speed. We need to close the loop between users, Ops and Dev. With CF, Dev and Ops can talk about the same things, the apps. Dev can be involved in the way their apps are performing, and can react quickly with new releases. Ops can better understand the underlying architecture, spot bottlenecks and talk to Dev in a more proactive way. Actually, companies could leverage CF to support the Amazon model of "you build it, you run it".

CF gives a framework to operate the latest best practices

A lot of things have been said about the benefits of 12 factor app and microservices. CF actually gives the framework where these principles can be applied in a repeatable way. It is pretty impressive to see that developers using CF are applying these principles sometimes not even knowing about them, just because CF is guiding them.

CF enables companies to catch up

And because the best practices are at the core of the development and deployment model, companies can deploy CF in their own data centers or through a PaaS provider to quickly catch up. Deploying at scale, in a reliable and repeatable way, is no longer reserved to large Web companies.

CF enables companies to simplify processes and reduce time-to-market

Several companies at the summit mentioned that they were able to drastically reduce the processes and time-to-market from months to days (AllSate, Humana). This is really a revolution in these companies where the process could have taken up to 100 days just to get the environment setup. Once the upper management agreed to the new process, results can be delivered very fast. It also pushes the ball to the app designers, marketing and architects to come up with ideas for their business, knowing that developing and deploying is no longer a barrier.

CF helps avoiding PaaS vendor lock-in

CF provides a framework that can be deployed directly in your own data centers or an IaaS such as AWS, or accessed through PaaS providers (IBM, GE, EMC, SAP, Huawei...). So on paper it could avoid vendor lock-in. However, these environments will still offer different services on top of CF, and it is unclear how you could migrate from one provider to the other. However, some of the concepts and tools will be the same so that an active community can be leveraged.

CF enables continuous innovation and continuous disruption...

When you put all these benefits together, you can see that CF enables continuous innovation by liberating people and removing lots of the existing barriers. If companies can take advantage of these new tools and processes, this could mean continuous disruption.

However, there are some challenges going ahead. I believe it will only work for companies willing to change their processes and for developers capable of leaving their comfort zone, learning new tools and languages, and following an ecosystem that is moving very fast. So for companies and developers alike, it is time to seize the opportunity.

What is DOcloud?

DOCloud is the short name of the IBM Decision Optimization on Cloud service. This service lets you solve CPLEX and OPL problems on the Cloud. You can access the interactive service called DropSolve or you can use use the API to integrate the service into your application.

CPLEX is a mathematical programming solver for linear programming, mixed integer programming, and quadratic programming. CPLEX is known as the leader and most efficient solver of its kind. It is a prescriptive analytics tool that helps you take better decision about your business. OPL is a modelling language that facilitates the use of CPLEX. These tools used to be available on-premise only, with a nice framework to build application called Decision Optimization Center.

The good news is that you can now access these tools on-line!

DropSolve is a simple web interface where you can literally drag and drop your model and data files, and they will be executed on IBM servers. If you want to build an application, you can use the API. The API is pretty simple: you create a job with the model and data, then you submit and monitor it, and finally you get the results once it is completed. There is a developer community where you can access to the API documentation, the examples and a forum.

To help you get started, I have created a YouTube playlist where you will find tutorials. The blog "IT Best Kept Secret Is Optimization" is also a must-read about optimization and DOcloud.

How to retrieve incrementally a time series using a REST API?

In this post, I would like to propose a REST API pattern to retrieve incrementally a time series.

Let's take the example of a system generating a time series such as price change, log events, machine status change etc. This system is able to send a sequence of data points using a messaging protocol, but as with many messaging protocols, the delivery and the order of delivery are not guaranteed. The goal is to propose a service that will record these messages and provide a REST API to incrementally access existing data points or new data points when they are available and in order.

First of all, we can easily record these messages in a data store, let's say using JSON format and in a MongoDB collection or in a CouchDB database. We can also notice that messages should be ordered and assigned a sequence number at the source. This way, it is possible to tell if 2 messages are contiguous or if there are one or more messages not yet received in between. So the first part of the service is about subscribing to the data stream, and storing the messages as they come.

Then, the second part of the service is to provide a REST API to access the time series (no Websockets or long polling for now). Well, the REST API can be very simple as this one:

   GET https://www.myservice.com/timeseries/TS?start=X

where TS is the name of the time series and X the sequence index at which we need to start returning the data points. The returned data could be of this form with X=10:

[ 
  { "index" : 10,
    "date" : 1423259728987,
    "value" : 10},
  { "index" : 11,
    "date" : 1423259730094,
    "value" : 42},
  ...
]

However, there are a couple of things to consider:

• The list of data points can be very large, and could cause out-of-memory crashes on the client or on the server depending on the implementation.
• Data points are received asynchronously, and there is no guarantee that the returned list will contain a continuous sequence.
• Some data point may be lost and will not be delivered, and again there is no guarantee that the returned list will contain a continuous sequence.
• Finally, the data stream may have ended, so that no more data points will be available and we need to indicate this.

With this in mind, I would like to propose the following design:

• There must be a server side limit on the number of data points returned at a time. This limit can be a default and potentially a lower limit could be specified by the client, but the important thing is to define a reasonable maximum limit to enforce. With this approach, the client can iterate over the data points, and specify the next start index as the last index + 1
• As some data points may be received out of order, gaps in the sequence can happen. In this case, the returned list should stop at the first gap detected. So if we recorded data points X, X+1, X+2, X+4, the call would return only X, X+1 and X+2.
• But some data point may be lost, and we need to set a maximum delay when we will consider that the data point will not be returned. If we continue the previous example, the next call will be with X=X+3. If the elapsed time between X+4 and the current time is over a maximum delay, we should assume X+3 lost, and we can return a fake data point with an attribute missing set to true, along with the rest of the sequence.
• Finally, if we know there is no more data points because the stream ended, we can indicate this by returning a fake data item with an attribute stop set to true,

In conclusion, the client can poll the server until it receives the stop flag. At each call it will receive no more than the maximum block size defined. It can specify the next call by adding 1 to the last index received. The API also guarantees that the data items are returned in order, and that if a data item is not available after some maximum delay, it will be returned and flagged as missing along with the rest of the data points. I believe this approach can be of a general interest.