This forum has been archived. Please start a new discussion on GitHub.

IceStorm - durable subscription

With most message queue systems there's a concept of "durable" subscription where subscribers can be taken down and when it comes back up, it'll get the messages that it misses while it was down.

Is there an equivalent feature in IceStorm?



  • xdm
    xdm La Coruña, Spain

    IceStorm currently doesn't provide such a feature.

    The current behavior is that IceStorm discarded a messages as soon as it is delivered, if IceStorm encounters a failure while attempting to deliver a message to a subscriber, the subscriber is immediately unsubscribed from the topic on which the message was published.

    See here

    So you will need to implement that your self, it could be a new service that use IceStorm to handle the message delivery.

    If you have an commercial interest in that feature, please contact to

  • benoit
    benoit Rennes, France

    Note that the subscriber is automatically un-subscribed from IceStorm if IceStorm gets an Ice::ObjectNotExistException or Ice::NotRegisteredException exception when invoking on the subscriber (these are considered hard failures). For other exceptions, IceStorm uses the retryCount QoS to figure out when the subscriber should be unsubscribed. In any case, while a subscriber is un-reachable the events are discarded and the delivery resumes only once the subscriber is available again. For more information on this retryCount QoS, see here in the manual.

    Perhaps you can detail a little why you need event queuing while the subscriber is down? An alternative solution to this might be to use per-subscriber publishers. Basically, the idea is that each time a subscriber comes online, it contacts the publisher to retrieve the latest state of the publisher. The publisher sends this information through the per-subscriber publisher object. Once it has retrieved the latest state, the subscriber can start receiving and processing again updates. This works well when using IceStorm to implement the observer pattern and avoids having to queue updates in the IceStorm service. The manual describes an example here. The C++ demo/IceStorm/counter demo also demonstrates how to use it.

  • Thanks for the replies.

    I don't currently have any specific examples. I'm comparing different systems and this is just one feature that another system provides.

  • matthew
    matthew NL, Canada
    The important thing to remember here is that there is no free lunch. In my experience in implementing exactly such as system (a notification system with durable events) there is a very big cost which you pay for such durability in terms of raw system throughput. In particular, if you want guarantees that the messages won't be lost, or even worse the messages respect transaction boundaries (in the case of distributed transactions) throughput can drop from thousands of events per second, to much much lower.

    In most cases there is a better way to design your system (as Benoit pointed out) where you don't have to pay such a high cost.

    Another thing to note is that IceStorm is specifically not a message queue. Its a publish/subscribe system. You should make sure you are comparing apples to apples.