Status update on SQLbusRT

I'll be on holidays from the 25th of August till the 5th of September. Therefore, I thought it would be a good idea to give an update on the current status of SQLbusRT.

Coding has almost finished. With almost finished, I mean it is almost ready for the first test runs. It is still a very simple implementation. It will give me some baseline figures when I execute my first performance tests, but it does not have all the planned functionality yet.

I'm not publishing the code yet. I want to have the code reviewed by some collegues first, and perhaps it's better to wait for the results of the first test runs to see whether the taken approach is a good one.

I've finished setting up a network of computers to run my tests on. All computers run linux, with the possibility of choosing a patched or an unpatched kernel on boot. The patched kernel offers preemption. This will be used to give real time priority to the incoming data from the network adapters. I'll give more detailed information on this after my holidays.

The 6th of September I'll be running those first test runs. As soon as some useful data has been extracted, it'll be posted here.

Handling SQL requests in SQLbusRT

While implementing the first test version of SQLbusRT, I faced a problem on how to communicate SQL requests and query results using ORTE as a medium.

Usually when doing an SQL request, a connection is set up between a client and an SQL server. The client sends its SQL request over this connection, and as an answer it receives the query results over the same channel. Unfortunately, in the bus architecture I have in mind things aren't that simple...

Remind the fact that SQLbusRT is built on top of a publish/subscribe model. Ideally speaking clients should be subscribers, and the database server should be a publisher. However, this introduces a problem for both instances:

• How does the publisher know what to publish?
• What should the subscriber subscribe to?

I already thought of a solution before I started implementing. This solution was even visible in my diagrams (see my previous posts) . The idea was to make a client act as a publisher to publish its request.

This idea however was not complete. While struggling with the question on how to actually implement this using publish/subscribe, I came up with the following:

• The request handler has a subscriptions for SQL requests
• The client publishes an SQL request
• The client creates a subscription for information from the request handler
  
• The request handler sends information on which data source to subscribe to
• The request handler sets up a publication for the new request
  
• The client subscribes to the correct data source

The following activity diagram shows how this communication is set up. It shows what data is present in the different messages. (Actual data, 'topic' and 'request', as required in RTPS.)
It also shows at what time subscriptions are being created in handlers and clients.

(Click to enlarge)

Note: The SQLrequest as you see it in the diagram is an object, containing the clientID, requestID and the actual SQL query. The response is also an object, containing the publicationID and requestID.

Working this way has a great benefit when multiple clients are making the same periodic or triggered request. The data only has to be fetched from the database once and will be distributed automatically, by letting the requesting clients subscribe to the same publication.

Development update

Since it is two weeks ago since my last post, I thought it would be a good idea to give a quick update on the development status.

At present, I've tackled all include and linking problems I faced (C++ is rather new to me). The insertion interface is ready, and the API for clients and sensors is near completion.
Last things remaining are the selection interface, an example client and an example sensor.

In a couple of days I hope to publish a package containing the first version.

Development picked up again

Some weeks back I posted that the development of SQLbusRT had started. After some days this task had been postponed to reconsider the architectural design and to do some more theoretical research.

The version I started was programmed in C, which seemed to be a logical choice since there's a good ORTE API in C.

Yesterday I restarted the coding, now with C++ as the language. It's still possible to use the C API for ORTE, but it also allows me to have an object oriented approach.

The following components will be present in the first release:

• A tiny API for writing simple clients and sensors
• An insertion interface, which writes the sensor data to the database
• A selection interface, which handles query requests from clients
• A small example client and sensor

As soon as a package becomes available, I will post a message here on how to retrieve it.

Performance, Reliability and Scalability

It is always nice to be able to say a system "performs well", is very "reliable" or is "scalable". But what do these terms actually mean?

One of the primary goals of my research on SQLbusRT is finding formulas which can predict the performance, reliability and scalability of the system under development, looking at the scenario it is implemented in.

Ofcourse saying "good performance", "very reliable" and "yes it is scalable" does not suffice. The formulas should provide a meaningful outcome, which is comparable to other outcomes.

I might not be clear to everyone. Let me give an example, using something we always like to talk about: the weather.

We can say the weather is good or bad today, but that is just a personal opinion. If we want to compare todays weather with the weather of tomorrow for instance, we need numbers. We can give useful information on the weather by giving:

• The temperature in Degrees Celcius or Fahrenheit (min, max, current)
  
• The humidity in %
  
• The wind speed in km/hour or miles/hour
  
• ...

The same way I would like to say something about the Performance, Reliability and Scalability of the system.

The more I read about it on the net, the more questions come in mind. I already found some scientific articles on the web which might help me, but I'm still looking for some more information. Especially scalability is often rather vague.

If someone knows any clear information on this matter, you're heartily invited to post a comment on this issue. You can help me a lot by doing so. All help is very much appreciated.

PS. I know that performance, reliability and scalability might have different meanings in different situations. Therefore, it is good to know that the system under development is a publish subscribe model with history added. You can find images explaining the architecture in my previous posts. Once again, all help is very much appreciated.

Clarified architecture

Some weeks back I posted two diagrams with the architecture for SQLbusRT. One was for one database instance, and one was for multiple databases.

After reading a bit on about the triggers and events in MySQL 5.x, I decided to revise the architecture slightly. In my previous architecture, I had taken event handling outside of the database, but since MySQL supports triggers and events, there is no need to do this anymore.

I've not only taken out the event handling, I've also enhanced the readability of the diagram. To sum it up, these changes have been made:

• Event handling is now handled inside the database, and has therefore been taken out of the diagram
  
• The edges are now labeled for clarification
  
• Grey boxes have been added to show the process boundaries; every box is one single process
  

These changes result in the following diagram (click to enlarge):
One thing about the processes is important to understand: ORTE didn't get it's own grey process box, because the amount of processes running for ORTE depends on the number of systems connected to the ORTE bus.

For every network interface connected to the ORTE bus, one "ORTE manager" is started. This manager handles the connections between the publishers and the subscribers. E.g., when all processes are running on one machine, then one ORTE manager will be running. Distributing the processes over multiple machines will increase the number of ORTE managers.

Planet MySQL

This blog has been added to the MySQL developer zone!

You can reach it by going to the MySQL website and clicking Developer Zone -> blogs. This brings you to the Planet MySQL, which now contains this blog.

You can also go there directly by following this link: Planet MySQL

New presentation

Last Friday, my mentor from the university came to visit Imtech ICT. To explain him what the project is all about, I gave a short presentation. The presentation contains some of the elements that are also present in the presentation that I posted on the 29th of May, but this new presentation focusses on the project instead of the product.

I'm sorry English readers, this presentation is in Dutch.

The presentation can be downloaded here: Kennismaking

Development has started

The development of a prototype of SQLbusRT has started.

This first version will contain an implementation of all the components in the current architectural design. However, these components will not contain much functionality yet. The interfaces will be designed, but the messages being sent between the components will mostly be hardcoded.

This first version will be used to do some first measurements on the real-time characteristics of the architecture. As soon as the coding has finished, a link to the package will be posted here.

After doing the first measurements, the results will be posted here as well.

If you are interested in specific measurements, please let me know. I might include them in my research.

Presentation

This morning I held a short presentation to share my vision on the project. The handouts for this presentation with the title "SQLbusRT: A first glance" are available at our SourceForge project website.

You can download these handouts directly by clicking this link: SQLbusRT: A first glance