IceStorm: Persistent messages

stephan

Hi there!

I've already asked about a feature that allows to persistently store icestorm messages quite some time ago.
The goal would be to have a guaranteed delivery of messages even if one of the subscribers is temporarily unavailable.

At the time of my last question, you did not intend to implement it into Ice. Is this still valid or are you about to implement something like that?

Just in case you aren't, I've taken a look into the IceStorm code and have a few questions about it.
Imho, the place to integrate is OnewaySubscriber.cpp . Is this correct?

I found a question called persistent(). Is it a placeholder for buffering messages in case the underlying subscriber object is temporarily not available?

For implementing persistency, I think, one would have to change the publish() code and store the message that should be sent to _obj in case of an exception, right?

Also, when the _obj is available again, activate() should push all pending messages (using publish() ?) to the _obj object.

Are these hypotheses correct or did I miss some details?

Thanks in advance for taking care e:)

regs,

Stephan

matthew

Re: IceStorm: Persistent messages

Originally posted by stephan
Hi there!

I've already asked about a feature that allows to persistently store icestorm messages quite some time ago.
The goal would be to have a guaranteed delivery of messages even if one of the subscribers is temporarily unavailable.

At the time of my last question, you did not intend to implement it into Ice. Is this still valid or are you about to implement something like that?

We have no immediate plans to do this, no.

Just in case you aren't, I've taken a look into the IceStorm code and have a few questions about it.
Imho, the place to integrate is OnewaySubscriber.cpp . Is this correct?

I found a question called persistent(). Is it a placeholder for buffering messages in case the underlying subscriber object is temporarily not available?

For implementing persistency, I think, one would have to change the publish() code and store the message that should be sent to _obj in case of an exception, right?

Also, when the _obj is available again, activate() should push all pending messages (using publish() ?) to the _obj object.

Are these hypotheses correct or did I miss some details?

Thanks in advance for taking care e:)

regs,

Stephan

If you take a very simple approach I would probably create another type of subscriber object PersistentSubscriber and then implement the subscriber interface. You could use an approach similar to the one that you've detailed above.

The persistent() method that you mention is used for another purpose -- namely whether the subscriber persists across multiple runs of the icestorm service, not whether the subscriber has persistent queue state.

Unless you are careful you will create an implementation that will not scale very well. In the case that a persistent subscriber is unreachable you would have to persist this message into some form of backing store (you could use Freeze for this) unless you are prepared for the messages to be lost over multiple runs of the icestorm service.

There are lots of tricky issues to deal with. For example, if you want to save memory and disk space then you'll want to ensure that you only store one copy of this data (not one copy for each offline subscriber). The current icestorm service creates an Event object for each message that arrives, and then directly distributes this object to each connected subscriber object. You could use something similar to this and store it into a shared topic database, and then reference this event from the queues of each persistent subscriber.

Regards, Matthew