Final presentation on SQLbusRT

After quite some months of silence, here is finally a new post on SQLbusRT.

The SQLbusRT project was actually the final project of the Master program I was in, and the last months have been very hectic. But, luckily I've succeeded, because on the 23rd of February I received my Master degree!

Click here for the slides of my final presentation. Unfortunately for some of you, these slides are in Dutch.

There is still a lot of work to be done in the SQLbusRT project. We would like to investigate the use of temporal and/or active databases, efficient query handling, request priority, matching of similar queries and much more. For this we are looking for one or more new Master students willing to take SQLbusRT to the next level.

If you are a Dutch Bachelor or Master student, and you are interested in a final project on SQLbusRT (at Imtech ICT in Amersfoort), please leave a comment with your contact info. You can also drop me a line at bram dot smulders {at} imtech dot nl.

YAP: Yet Another Presentation (on SQLbusRT)

Friday the 3th of November I presented my work at the University of Twente to my fellow students and my supervisors. The slides of this presentation are available through this link:
Presentation at University of Twente, 3th of November 2006

All students and supervisors present that day are encouraged to post their comments.

New test strategy for SQLbusRT

The test results that I have generated before (see my previous post) did not really help me in reaching my project goal, which is to make it possible to predict SQLbusRT's performance, reliability and scalability when it is used in an application.

In my new strategy I'm gonna take one step at a time to put the different components of SQLbusRT to the test.

My first upcoming test will involve sending messages back and forth between two processes, using the ORTE communication bus. This way, I can really measure ORTE's performance and reliability alone, without influence of MySQL. I will perform this test in different scenario's which will let me determine the scalability as well.

After having processed the results of these tests, I will include MySQL. Making these separate tests will let me determine the influence of ORTE and MySQL separately.

The new test results will of course be posted here again as soon as they are processed.

Presenting the first SQLbusRT test results

Here it is, a presentation after my first tests. Open the slides by clicking here. (Or right click, save as...).

The presentation is not exhaustive if it comes to the test results, so if anyone is interested in the log files themselves, you can drop me an email at bram.smulders-at-imtech.nl or post a comment with your email address.

SQLbusRT: First test version is running!

Today I have some good news to announce! Finally, after quite some struggles and some disappointments because of 'optimistic planning', I can tell you that the first version is working.

It is a testing version which has the following functionality:
- An example sensor publishes random values on the bus;
- A insertion interface reads all sensor data on the bus (it is ready to receive data from multiple sensors) and writes it to the database. It creates tables for every new sensor;
- A selection interface listens to SQL requests and creates data sources which publish data on the bus at a specified interval;
- An example client publishes it's request on the bus and after receiving a data source ID, subscribes to this data source. It will from thereon receive the result periodically.

I will add some logging to the code now so I can extract some meaningful performance information.

I will present the performance measures to my colleagues next Monday. Afterwards, I will put the slides online.

Back to work (and more on ORTE)

After coming back from holidays on the 6th of September, I've been able to work on the project with a refreshed mind again!

I expected to have finished coding by now, but after putting all the components together, I ran into an occasional 'segmentation fault'. I've been looking for the problem, but only with the help from one of my colleagues I was able to address the causes.

For the first cause, I should explain a little bit about how ORTE handles issues:
As I explained before, ORTE works with 'publishers' and 'subscribers'. ORTE periodically invokes a callback function on the publisher side, which is meant to prepare the data for sending in a memory buffer. When the callback function finishes, ORTE reads the buffer and copies it to a memory buffer on the subscriber side. It then invokes the subscriber callback function which is meant to process the incoming data.

So what went wrong in my implementation?
My request handler acts as a subscriber for requests. Whenever a request comes in, it behaves as a publisher to notify the requesting client on the data source ID. (You can find this scenario in the diagram added to my post of the 3th of August.)
I used the request handler's subscriber callback function to immediately create a publication of the data source ID. And that was the problem! ORTE only allows creation of publications in the main thread. When you try to create it in the subscriber callback function, it will be handled in another thread, causing it to fail.
I have already fixed this problem by placing the creation of the publication in the main thread, controlling it with the use of semaphores.

What about the second cause?
The second cause for the segmentation faults was caused by my enthusiasm to free memory as soon as possible.
After making the request handler send the data source ID, I immediately freed the memory used by this publication. The C code looks like this:

//p is the publication handle
ORTEPublicationSend(p); //sending the issue
ORTEPublicationDestroy(p); //destroy the publication handle

I expected the ORTEPublicationSend function to be locking the thread until the entire publication had been finished, but it turns out to be a false assumption. With the code above, I was destroying the publication handle before the publication was finished completely, causing a segmentation fault.

This issue has been fixed with a workaround for now.

What coding is still to be done?
For my first measurements, I have to set up all the data sources in different threads. Using the metaphors fixing one issue has introduced a new one. It prevents the data sources from sending data.

After this, I will give my code a review, and I will invite a colleague to review my code critically.

Spam has arrived!

Unfortunately, after having my blog on the net for about 4 months, spam bots have found my blog. To prevent any further bot comments to appear on my blog, I've switched on 'word verification' for comments.

Please don't let it hold you back from posting comments on my blog. All your (non-spam) input is very much appreciated!

Status update on SQLbusRT

I'll be on holidays from the 25th of August till the 5th of September. Therefore, I thought it would be a good idea to give an update on the current status of SQLbusRT.

Coding has almost finished. With almost finished, I mean it is almost ready for the first test runs. It is still a very simple implementation. It will give me some baseline figures when I execute my first performance tests, but it does not have all the planned functionality yet.

I'm not publishing the code yet. I want to have the code reviewed by some collegues first, and perhaps it's better to wait for the results of the first test runs to see whether the taken approach is a good one.

I've finished setting up a network of computers to run my tests on. All computers run linux, with the possibility of choosing a patched or an unpatched kernel on boot. The patched kernel offers preemption. This will be used to give real time priority to the incoming data from the network adapters. I'll give more detailed information on this after my holidays.

The 6th of September I'll be running those first test runs. As soon as some useful data has been extracted, it'll be posted here.

Handling SQL requests in SQLbusRT

While implementing the first test version of SQLbusRT, I faced a problem on how to communicate SQL requests and query results using ORTE as a medium.

Usually when doing an SQL request, a connection is set up between a client and an SQL server. The client sends its SQL request over this connection, and as an answer it receives the query results over the same channel. Unfortunately, in the bus architecture I have in mind things aren't that simple...

Remind the fact that SQLbusRT is built on top of a publish/subscribe model. Ideally speaking clients should be subscribers, and the database server should be a publisher. However, this introduces a problem for both instances:

• How does the publisher know what to publish?
• What should the subscriber subscribe to?

I already thought of a solution before I started implementing. This solution was even visible in my diagrams (see my previous posts) . The idea was to make a client act as a publisher to publish its request.

This idea however was not complete. While struggling with the question on how to actually implement this using publish/subscribe, I came up with the following:

• The request handler has a subscriptions for SQL requests
• The client publishes an SQL request
• The client creates a subscription for information from the request handler
  
• The request handler sends information on which data source to subscribe to
• The request handler sets up a publication for the new request
  
• The client subscribes to the correct data source

The following activity diagram shows how this communication is set up. It shows what data is present in the different messages. (Actual data, 'topic' and 'request', as required in RTPS.)
It also shows at what time subscriptions are being created in handlers and clients.

(Click to enlarge)

Note: The SQLrequest as you see it in the diagram is an object, containing the clientID, requestID and the actual SQL query. The response is also an object, containing the publicationID and requestID.

Working this way has a great benefit when multiple clients are making the same periodic or triggered request. The data only has to be fetched from the database once and will be distributed automatically, by letting the requesting clients subscribe to the same publication.

Development update

Since it is two weeks ago since my last post, I thought it would be a good idea to give a quick update on the development status.

At present, I've tackled all include and linking problems I faced (C++ is rather new to me). The insertion interface is ready, and the API for clients and sensors is near completion.
Last things remaining are the selection interface, an example client and an example sensor.

In a couple of days I hope to publish a package containing the first version.