Development picked up again

Some weeks back I posted that the development of SQLbusRT had started. After some days this task had been postponed to reconsider the architectural design and to do some more theoretical research.

The version I started was programmed in C, which seemed to be a logical choice since there's a good ORTE API in C.

Yesterday I restarted the coding, now with C++ as the language. It's still possible to use the C API for ORTE, but it also allows me to have an object oriented approach.

The following components will be present in the first release:

• A tiny API for writing simple clients and sensors
• An insertion interface, which writes the sensor data to the database
• A selection interface, which handles query requests from clients
• A small example client and sensor

As soon as a package becomes available, I will post a message here on how to retrieve it.

Performance, Reliability and Scalability

It is always nice to be able to say a system "performs well", is very "reliable" or is "scalable". But what do these terms actually mean?

One of the primary goals of my research on SQLbusRT is finding formulas which can predict the performance, reliability and scalability of the system under development, looking at the scenario it is implemented in.

Ofcourse saying "good performance", "very reliable" and "yes it is scalable" does not suffice. The formulas should provide a meaningful outcome, which is comparable to other outcomes.

I might not be clear to everyone. Let me give an example, using something we always like to talk about: the weather.

We can say the weather is good or bad today, but that is just a personal opinion. If we want to compare todays weather with the weather of tomorrow for instance, we need numbers. We can give useful information on the weather by giving:

• The temperature in Degrees Celcius or Fahrenheit (min, max, current)
  
• The humidity in %
  
• The wind speed in km/hour or miles/hour
  
• ...

The same way I would like to say something about the Performance, Reliability and Scalability of the system.

The more I read about it on the net, the more questions come in mind. I already found some scientific articles on the web which might help me, but I'm still looking for some more information. Especially scalability is often rather vague.

If someone knows any clear information on this matter, you're heartily invited to post a comment on this issue. You can help me a lot by doing so. All help is very much appreciated.

PS. I know that performance, reliability and scalability might have different meanings in different situations. Therefore, it is good to know that the system under development is a publish subscribe model with history added. You can find images explaining the architecture in my previous posts. Once again, all help is very much appreciated.

Clarified architecture

Some weeks back I posted two diagrams with the architecture for SQLbusRT. One was for one database instance, and one was for multiple databases.

After reading a bit on about the triggers and events in MySQL 5.x, I decided to revise the architecture slightly. In my previous architecture, I had taken event handling outside of the database, but since MySQL supports triggers and events, there is no need to do this anymore.

I've not only taken out the event handling, I've also enhanced the readability of the diagram. To sum it up, these changes have been made:

• Event handling is now handled inside the database, and has therefore been taken out of the diagram
  
• The edges are now labeled for clarification
  
• Grey boxes have been added to show the process boundaries; every box is one single process
  

These changes result in the following diagram (click to enlarge):
One thing about the processes is important to understand: ORTE didn't get it's own grey process box, because the amount of processes running for ORTE depends on the number of systems connected to the ORTE bus.

For every network interface connected to the ORTE bus, one "ORTE manager" is started. This manager handles the connections between the publishers and the subscribers. E.g., when all processes are running on one machine, then one ORTE manager will be running. Distributing the processes over multiple machines will increase the number of ORTE managers.

Planet MySQL

This blog has been added to the MySQL developer zone!

You can reach it by going to the MySQL website and clicking Developer Zone -> blogs. This brings you to the Planet MySQL, which now contains this blog.

You can also go there directly by following this link: Planet MySQL

New presentation

Last Friday, my mentor from the university came to visit Imtech ICT. To explain him what the project is all about, I gave a short presentation. The presentation contains some of the elements that are also present in the presentation that I posted on the 29th of May, but this new presentation focusses on the project instead of the product.

I'm sorry English readers, this presentation is in Dutch.

The presentation can be downloaded here: Kennismaking

Development has started

The development of a prototype of SQLbusRT has started.

This first version will contain an implementation of all the components in the current architectural design. However, these components will not contain much functionality yet. The interfaces will be designed, but the messages being sent between the components will mostly be hardcoded.

This first version will be used to do some first measurements on the real-time characteristics of the architecture. As soon as the coding has finished, a link to the package will be posted here.

After doing the first measurements, the results will be posted here as well.

If you are interested in specific measurements, please let me know. I might include them in my research.

Presentation

This morning I held a short presentation to share my vision on the project. The handouts for this presentation with the title "SQLbusRT: A first glance" are available at our SourceForge project website.

You can download these handouts directly by clicking this link: SQLbusRT: A first glance

Support for multiple DB instances

As I mentioned in my previous post, I would change the architecture to support multiple DB instances. In fact, the change was fairly simple, as you can see in the following drawing:

As you can see, I added a copy of the complete set of components which interact with the database. In the drawing two databases are connected to the bus, but copying the components can be repeated multiple times to allow more db instances to connect to the bus.

So what are the changes in the scenario?
In the previous design, only one db subscriber requested the sensor data to put it in the database. Again, only one one db publisher was present to publish the query results onto the bus.

In this new design, multiple instances of the db subscriber are reading the sensor data from the bus. They are responsible to pass the information on to their own database that is connected to them. Every database instance has its own publisher to make the data available on the bus again. This means that in normal operation, duplicate data is sent to the bus. However, each db publisher has a different strength on the ORTE bus. Therefore, clients will only receive the data once.

In case a database fails to produce data (for instance, the db connection is broken, or the db is too busy and therefore is not able to produce the data in time) the data from the publisher with the second highest strength is read by the clients.

The great benefit of this design is that no reconnect to a database duplicate is necessary when the master database fails. The second (and third, fourth...) database instance is already up and running and even publishing its data.

Conceptual architecture

In the past week a bit of collaboration with colleagues resulted in the following conceptual architecture:

You can look at the project website at sourceforge for an explanation of the different components in this diagram.

Soon (perhaps today) I'll post a new conceptual architecture, which lets multiple databases connect to the bus.

Feel free to comment on this architecture!

Welcome!

Welcome to my SQLbusRT blog!

My name is Bram Smulders. I recently started my final project for Imtech ICT Technical Systems in Amersfoort, the Netherlands, to finish my Masters degree at the University of Twente.

This blog will be used to keep all interested parties informed on the progress of this project.

So what is SQLbusRT?
SQLbusRT is a bus architecture with database connectivity which lets multiple publishers and subscribers exchange data meeting real time constraints. The benefit of having the database connectivity is that not only new data can be read by subscribers, also history data can be retrieved from the database, still meeting the real time constraints.
Currently the project is in a conceptual phase.

Information on this project
Besides this blog, there's also a website at SourceForge to keep you up to date on the project. You can find it at: http://sqlbusrt.sourceforge.net
The sourceforge website is the main website for this project. You'll find all the information available on this project right there. This blog however is to give you quick updates and to share thoughts with all interested parties. You can (read: you're heartily invited to) post your comments, share your thoughts and give all kinds of ideas related to the project.