Solution Variation: (Nearly) Real-Time Data

When trading equities or fixed-income products on a market, you will also want to retrieve realtime data sets. By “real-time” data sets, I mean nearly real-time data sets, as people building truly real-time applications might take offense at how loosely I define “real-time.” From a personal perspective, I believe that real-time data is not possible with Ajax, as there are simply too many layers of software. What is possible is nearly real-time data, with about half a second or so of lag (of course, this is based on my own experiences it’s by no means definitive). In terms of most trading applications, not including scalping, nearly real-time is good enough. Real-time data is a variation of the article we’ve already seen, where a real-time task is similar to a long-running task. A long-running task that generates data is like a real-time task, in that both generate data whenever they can. The difference between a long-running task and a real-time task is that the data generated by a real-time task has a source beyond the task (e.g., the stock market, a machine running process, etc.). Another difference is that usually a long-running task will have an answer, whereas a real-time task has no answer just data that keeps being generated.

The mismatch of what each client sees as data can be desired or it can be a problem. For example, say that you are controlling a machine process that creates foam car seats. Each seat takes about 5 minutes to create, and so your Ajax application will store about 7 minutes of data. The window of time for your application is big enough to include one cycle of the item being watched.

Now let’s turn our attention to the stock market. What would be an adequate window of time for watching a stock ticker? Would it be 1 minute, 10 minutes, a few hours, days, weeks, or years? And depending on the window, what is the length of each individual graph marker? Is it 1 second, 5 seconds, hours, days? Again, we don’t know, and we have a problem in that a user could ask for a graph for each second for the past year. The data for such a graph is enormous. In real time on the client side, you can only reasonably expect to keep track of data within a certain window of time. Keeping track of a window of time is acceptable if you are only interested in the window being presented. The reality is that you will want to know what happened in the past. Looking at the past data for a real-time task is called backfilling the data. In the case of the equity example, you need to include a backfill because the calculations performed on the stocks might not be accurate otherwise.

This then introduces a further complication: how do you manage a backfill and real-time task at the same time on the client? The answer is you don’t. An Ajax client is not capable of such complicated logic. The backfill and the real-time process need to be managed by the server. The client sees only a view of the data as if it were an infinite data source. When the real-time task starts, the time series data needs to have its metadata extracted from it. The metadata would then be used to determine where the real-time task adds data to the result set. So, for example, if you are tracking a real-time stock, the metadata would be the stock ticker and the time of the stock tick. If you started the real-time task at 10:00 a.m., then the first tick would be 10:00 a.m. If you are tracking stocks on the New York Stock Exchange, then your complete result set would span the times from 9:30 a.m. to 4:00 p.m., at least the official trading hours. For our example, that means the missing data are the ticks from 9:30 a.m. up to and including 9:59:59 a.m. The missing data is added to the result by executing a task that loads the historical data from an alternate data source. When the historical data has been added, the combination of the two data sets results in a complete and consistent data result set. Regardless of when or how end users see the data, all end users see the same data. You could come to the conclusion that when data is presented in a consistent format, you have switched the problem from a real-time data generator to a problem of generating a data set that just happens to require a complete day to calculate the results.

In implementation terms, the following rules of thumb are applied:

• You need to define what constitutes a complete result set. You need to define a complete result set in terms of days, hours, or some other quantifiable unit of measure. For example, you could use as the unit of measure an empty barrel that is filled.

• The data generated by the real-time task must be describable by its metadata, which can be used to create an index.

• When the real-time task is started, the data is added to the result set using an index.

• For situations where the real-time task is started at a later point in time, the missing data in the index is provided by a task that executes a backfill.

• A backfill task is executed whenever the real-time task misses data due for one reason or another. The backfill task also serves the purpose of ensuring that the data generated by the real-time task is consistent and accurate. From the perspective of the client, we see a complete result set with a large number of elements. The client decides at which index they want to start tracking the real-time generated data.

Article Summary This article’s focus was on building aWeb application that manipulates large or slow data sets. Keep the following points in mind:

• Large and slow data sets are dealt with using the same solution. The solution might have implementation variations, but there will always be a task-based approach and the use of a cache on the client or server.

• All results are composed of sets of data, where each individual result has a metadata element.

• The metadata element is used to uniquely identify an individual result. Based on the metadata, it is possible to algorithmically determine a URL to the data. There should be no metadata elements that can be confused with each other; however, a single metadata element descriptor can contain multiple individual results.

• Think of the metadata element as supporting your ability to implement set theory to select, display, or navigate data.

• Navigating large or slow data sets requires specialized navigation.

• The specialized navigation is based on users being able to quickly navigate the metadata, allowing users to get a quick and rough idea of what is contained in the data.

• The specialized navigation should involve the use of clickless navigation techniques as much as possible.

• The performance of the application is dependent on the implementation of the cache on the client and server.

• For most implementations of this article, you will need to tune the cache so that the cache can algorithmically determine what the end user needs to see next.

• The simplest cache for read-only type applications would be a single request client cache. For applications where the data changes regularly, an HTTP validation-based cache is needed. For any other application, the cache would be a combination of single request and HTTP validation.

• How long data is kept in the server and client caches depends on the nature of the application, and is a tunable parameter.

• A slow and large data set application distinguishes itself from a regular Web application in that the answer to a POST is retrieved using a separate GET.

• The POST will spawn a task where the task could be executing in another thread or in another process. The POST, having spawned the task, will in most cases return enough metadata information to uniquely identify the URL of the results.

• The results are stored in their own URL that may reference a cache. The results are generally only accessible using the GET and DELETE verbs. There may be ways to fine-tune the result set using subdirectories or query parameters.

• Do not delete results using a stack-based approach. The client in general would keep a cursor on the latest results or constantly download all of the results. This approach is preferred for stability and robustness.

• Real-time data is a variation of the slow and large data set article. The big difference with respect to a long-running task is that a real-time task will execute for as long as the monitored information is being tracked.

• Real-time data needs to be combined with a bigger picture of what a result set is. You want to convert the real-time data set into a bigger data set that includes a backfill, making it seem that the real-time data task is in fact a slow data task that happens to need a complete day or unit of time to complete its calculations.