Filtering in IceStorm?

SteveWampler

[I'm continuing an investigation of switching from CORBA to Ice...
hence the naive questions as I learn more about Ice...]

We currently rely heavily on events in our distributed processing. Under
CORBA, we've been using OmniNotify for our event service (because
our definition of an 'event service' is probably closer to a 'notification
service'). Events are posted by name and contain sets of attributes
(name/value pairs).

IceStorm provides an efficient general-purpose publish/subscribe
mechanism that meets most of our requirements and might be
adapted to act as the framework of our event service. I see two
approaches:

(1) A separate topic for each event name. This could lead to
*lots* of topics - is that a bad thing?

(2) An 'event' topic (possibly with federation to help with load
balancing). Then other topics could be used for other message
classes (log/debug messages, alarms, heartbeats, etc).

Regardless of the approach, however, there is one feature we rely on
with OmniNotify that seems to be missing in IceStorm: the ability
to subscribe to events using wildcarded event names (e.g. one
client subscribes to 'vsm.*', another to '*.camera.*', and a third
to 'vsm.camera.readout.noise'. While all three would see some
events in common, the first two would receive additional events
through the same subscription not visible to the third, and each
of the first two would see events not seen by the other.)

Now, I understand *why* IceStorm (I think) doesn't currently
support this - it treats messages as opaque, gaining in efficiency
and generality by doing so.

However, a reasonably efficient way to provide the above (and other)
message selection on a topic would be to provide application-specific
filters. In the above case, we would provide an event name filter that
would understand the event naming and wildcard conventions to
determine whether or now a message should pass through.

While this *could* be done in the client w/o affecting IceStorm, this would
effectively require all clients to subscribe to all events, which is undesirable
from both a network bandwidth perspective and cpu load point of view.

But, if we adopted approach (2) above, then (in theory), it would be possible
to attach these filters to the downstream connections from a topic, but within
IceStorm. While this increases the load on the IceStorm node it (a) is no worse
than the load on each client with a client-side solution and (b) possibly mitigated
through federation. It helps overall network bandwidth.

This would also have minimal impact when using IceStorm as it is currently
designed - just a quick check to discover that there is no filter would be added
to the processing.

So, am I out of my mind? Is these a reasonable approach? Has it be dicussed
before (I couldn't find it mentioned in the archives). Is there an existing
mechanism in Ice that would serve the same purpose that I'm missing? Would it
be hard to add? Would anyone else find it useful?

Thanks!
Steve

mes

Hi Steve,

If your subscribers need to subscribe to multiple events (and it sounds like they do), then I don't think approach #1 is very feasible (too much administrative work). The second approach sounds more appropriate, given what you've described.

Regarding filtering: We've had a few discussions about this, and our preferred strategy to address it involves the use of Ice request contexts. Specifically, publishers supply metadata in the request context, and subscribers could configure IceStorm to filter messages based on that metadata.

The advantage of this strategy is that it allows IceStorm to continue treating messages as opaque, because the request context is part of the message header, and not in the opaque message body. Consequently, filtering would not be affected by changes to Slice types or operation signatures.

We haven't implemented this technique, but we think it's a reasonable and technically sound solution.

Of course, since Ice makes it so easy to write an efficient event service, you could also elect to implement one yourself that meets your needs (and you could use the request context technique described above, if you like).

Another possibility would be to implement a filtering topic that you federate with IceStorm. This could even be collocated with the IceStorm service (via IceBox), although that's not currently as much of a performance advantage as you might think, for technical reasons I won't get into now.

As you can see, there are a number of ways to address your requirements. Let us know if you have any more questions.

SteveWampler

Thanks - that's useful! The filtering on metadata is how OmniNotify does it. In OmniNotify
it's adequate, but a bit inconvenient - you have to be careful laying out the metadata so
the way ON does filtering matches what you want filtered (for example, in the last project we
were limited to filtering on at most three fields, which wasn't quite as general as we'd have
liked.

You've given me some good info - thanks. (We're still in the investigation phase, so
I won't act on anything yet.;..)