multiple federated strom servers

When talking about fault tolerant event networks you generally have two different "types" of fault tolerance to consider:

• subscription fault tolerance, and
• event flow fault tolerance.

Subscription fault tolerance is essentially 'connection' fault tolerance. The objective is to make sure that if a node fails then there are one or more 'backups' that have the same connections or can recreate the connections to make sure that requests and events can still get through. The emphasis is on the structure of the event network, not the events themselves.

TAO's Fault Tolerant Real Time EventService implements this type of fault tolerance. TAO implements some elements of the CORBA FT spec which the fault tolerant versions of the TAO services use to create groups of replicas, manage fault notifications, etc. TAO currently doesn't address event fault tolerance.

There are three problems to solve to get this sort of fault tolerance with IceStorm:

1. Multiple IceStorm services need to be instantiated and configured with a topic.
2. The publishers need to be made aware of the aforementioned topics on the service replicas.
3. The subscribers need to subscribe to the aforementioned topics on the service replicas.

It is admittedly awkward to do this manually. Making IceStorm support subscription fault tolerance would involve providing facilities to make configuring and running replicas, publishers and subscribers easier and more convenient. In a "real world" deployment you would want things to be as transparent as possible for the subscribers and possibly also the publishers. For example, creating of a new topic shouldn't require programmatically contacting each replica in your application.

I am not a TAO specialist and stand to be corrected, but there appears to be a significant distinction between IceStorm and TAO here is that IceStorm implements link and topic persistence whereas TAO does not. The significance is that restarted IceStorm services do not need to fully recreate their connections by reconciling with the master before being ready to participate in event delivery.

However, IceStorm's topic persistence can also pose a bit of a problem here as it can be difficult to maintain consistency across replicas. Addressing database consistency across IceStorm is an issue. If the added efficiency and safety of having subscription persistence is not necessary we can simply remove this feature and manage subscription coordination on restart like TAO does.

We estimate it will take approximately 18 weeks to complete a subscription fault tolerance mechanism for IceStorm without subscription persistence. To keep subscription persistence and and ensure consistency across all topic replicas, we estimate an additonal 6 weeks.

Event fault tolerance implies subscription fault tolerance since it requires that connections be maintained so that events can continue to flow to their destinations. You want event tolerance if loss of events is an issue. With subscription only fault tolerance, any events that are currently queued by the current 'master' servers might be lost when a server fails. To acheive a fault tolerant event flow, events must somehow be replicated across the backup event networks so if a service goes down the events are not lost. This is much more involved.

As in the case of TAO, I am not a Spread expert so I stand to be corrected here, but the Spread Communication Toolkit provides a sort of event fault tolerance using a group communication model. Group memberships, message routing, etc. is dynamically and constantly coordinated among the members of the group. It looks like it is even possible to coordinate event delivery. The system appears to require a bit of configuration and the communication model is quite different than that of IceStorm. It would appear that you program to an API as opposed to a language mapping, which makes it seem more like a communication framework than middleware. However, it seems you can use a layer like their JMS4Spread implementation to insulate you a bit from the API.

We estimate it would take up to 16 additional weeks to complete fault tolerant event streams, with some variance depending on final requirements (e.g. event persistence, qualities of service, etc).

If you would like to sponsor any of these developments, please contact us at info@zeroc.com.

I'm sure this type of technology would be helpful for lots of types of projects as you point out. However, we only have limited time & resources -- so therefore we at this time generally only attempt to undertake those projects which have immediate commercial use. If you have a commercial use for this then please contact us at sales@zeroc.com to discuss the details.