tracing IceStorm

oroulet · February 2010

hi,

I have a reliabilty problem with IceStorm which is so bad that I am confident it is an error on my side :-) So I am looking for some help to debug it.

when running subsriber and publisher on the same machine as icestorm(and registry) it seems to work.
But when running on a wireless network, subscribers are sometimes not called at all for a few minutes, sometimes called one minute later in a burst and sometimes works for a few minutes.
The network has sometimes latency up to a second but it is not as bad as it seems to be when using icestorm. I will try with a wired network tomorrow to be sure.

Does anyone have an idea about what the problem can be? and how I can trace IceStorm in that situation ?

IceStorm is started through icebox with this simple configuration

<icegrid>
<application name="IceBoxApp">
<node name="Node1">
<icebox id="IceBoxServer" exe="icebox" activation="on-demand">
<service name="IceStorm" entry="IceStormService,33:createIceStorm">
<dbenv name="${service}"/>
<adapter name="${service}.TopicManager" >
<object identity="IceStorm/TopicManager"
type="::IceStorm::TopicManager"/>
</adapter>
<adapter name="${service}.Publish" />
<property name="${service}.Trace.Topic" value="1"/>
<property name="${service}.Trace.TopicManager" value="2"/>
<property name="${service}.Trace.Subscriber" value="1"/>
<property name="${service}.Transient" value="1"/>
</service>
</icebox>
</node>
</application>
</icegrid>

Thank you
Olivier

benoit · February 2010

Hi Olivier,

Which QoS do your subscribers use to subscribe to the topic? You could add some additional Ice tracing to see when IceStorm pushes the events to the subscribers by setting Ice.Trace.Network=2 and Ice.Trace.Protocol=2 in the IceStorm configuration (you could also eventually set Ice.Trace.Network=3 to see when data is written/received over the Ice connection).

Cheers,
Benoit.

oroulet · February 2010

Thank you for the tips,
It seems to work now.

There was several problems:

1. As you pointed out I used the default qos[retryCount] and in that unstable network some subscribers got disconnected. I suppose the fact that some subscriber restarted to work has to do with the $service.Send.Timeout or $service.Discard.Interval property. I am not sure I have understood the relation between those properties and the retryCount so I will need to check that.

2. Some subscribers did sometimes subscribe with the ip address from the wrong interface

Olivier

alerox · February 2010

Hi, I experienced the same problem just running IceStorm, along with the Subscriber and the Publisher of your very easy-to-understand clock sample. They both work fine, promptly and seamlessly when started on the same machine, but unfortunately when I try to move the same Subscriber on a different host, it starts receiving messages, remotely dispatched by the Publisher via IceStorm, only after 25-30 seconds. After this delay, it starts working fine again.
I read something about this in your Forum, but no solution has been provided (e.g. in http://www.zeroc.com/forums/help-center/2009-how-minimize-response-time.html where, I think, the problem is well discussed but not yet solved).
My question is: how can I avoid that initial delay? Are there some settings I can modify to start receiving messages from topics immediately?
I tried both .NET and Java version of the clock samples, editing the config.sub properly with the hostname first, and the IP address then, of the remote TopicManager. The version of Ice I’m using is the 3.3.1. I don't post any config file since they are all the ones you provide with your setup file itself.
Thanks in advance.
Regards.

bernard · February 2010

Hi Alessandro,

Welcome to our forums!

You could enable tracing as Benoit suggested earlier in this thread.

As Olivier pointed out, one possible issue with your configuration is that your Subscriber may be listening on interfaces that IceStorm cannot reach. This could/would introduce a long delay in the initial connection establishment. Check that you have "-h <IP/hostname of desired interface>" on all your .Endpoints configuration.

Best regards,
Bernard

alerox · February 2010

So, if I well understood, I need to set, inside config.icebox:

[...]
#
# Network Tracing
#
# 0 = no network tracing
# 1 = trace connection establishment and closure
# 2 = like 1, but more detailed
# 3 = like 2, but also trace data transfer
#
Ice.Trace.Network=2 (or 3)

#
# Protocol Tracing
#
# 0 = no protocol tracing
# 1 = trace protocol messages
#
Ice.Trace.Protocol=2

even if, in the latter case, the "2" value is not commented.
This has to be done in order to enable tracing.
But, in order to solve the problem of the delay, do I need to edit also sub.config, and in particular the property

Clock.Subscriber.Endpoints=tcp:udp

which is the default value, giving the specific ip of the subscriber? This setting is very useful, since IceStorm selects the first available port number, and permit to avoid to hard-code the (potencially unknown) IP number of the subscriber hosting machine.
Thank you very much for your prompt support.

bernard · February 2010

Hi Alessandro,

Each config.* file contains the properties of an Ice communicator:

config.icebox in this demo contains the properties for the communicator of the IceBox container, which typically does not do much (it hosts the IceBox service manager object, see http://www.zeroc.com/doc/Ice-3.3.1/manual/IceBox.45.5.html).

config.service contains the properties for the communicator of the IceStorm service, and that's where you want to set the tracing properties.

But, in order to solve the problem of the delay, do I need to edit also sub.config, and in particular the property

Clock.Subscriber.Endpoints=tcp:udp

which is the default value, giving the specific ip of the subscriber? This setting is very useful, since IceStorm selects the first available port number, and permit to avoid to hard-code the (potencially unknown) IP number of the subscriber hosting machine.

If you put your subscriber (or any Ice server for that matter) on a computer with multiple interfaces, and one or more of these interfaces are not reachable by all clients, then you need to specify which interface(s) your subscriber should listen on and publish in its proxies.

For example, let's say your computer has 2 network interfaces:
- interface 1, associated with IP 192.168.1.100
- interface 2, associated with IP 10.0.2.100

Clock.Subscriber.Endpoints=tcp:udp

means the object adapter Clock.Subscriber will listen on all interfaces for tcp and udp requests, using a system-provided port number.

And all proxies created by this object adapter will contain all these interfaces (except localhost) (and unless you configure Clock.Subscriber.PublishedEndpoints), i.e.

tcp -h 192.168.1.100 -p <OS-assigned-port>:tcp -h 10.0.2.100 -p <OS-assigned-port>:udp -h 192.168.1.100 -p <OS-assigned-port>:udp -h 10.0.2.100 -p <OS-assigned-port>.

Obviously Ice can't guess that 192.168.1.100 is ok while 10.0.2.100 is not (or vice versa), so you have to tell Ice which interface to use / publish in the proxies, for example:

Clock.Subscriber.Endpoints=tcp -h 192.168.1.100:udp -h 192.168.1.100

The port numbers are not affected, and will be assigned by the OS.

I hope this is clearer now.

Bernard

Archived

tracing IceStorm

Comments

Categories