Archived

This forum has been archived. Please start a new discussion on GitHub.

effect of IceStorm.Send.Timeout in Ice3.2

Hi,

I'm trying to understand better how IceStorm works in Ice3.2. To do this, I set up a few tests and I'm trying to break things. My current scenario is:

Host A:
- IceStorm
- One publisher

Host B:
- subscriber B

Host C:
- subscriber C

I've set the parameter:
IceStorm.Send.Timeout=1000
(remainder of IceStorm config file included at end of post).

Now, with the publisher on Host A publishing at a high frequency (~100Hz), I pull out the network cable from Host C for 10seconds, then stick it back in. Subscriber B continues to receive traffic as expected.
I would expect to see IceStorm cancel the subscription of subscriber C after one second (as described on page 1601 of the Ice manual).
I've got all IceStorm tracing turned on, but it doesn't bitch about anything.
Instead what I see is:
- subscriber C stops receiving traffic when the cable is unplugged (of course)
- when the cable is reconnected, subscriber C starts receiving traffic again (it receives the entire backlog of data from when the cable was unplugged, then all future new traffic).

Am I misunderstanding something here?


Thanks very much,

Alex


icebox config file below:
IceBox.ServiceManager.Endpoints=tcp -p 9998
IceBox.InheritProperties=1

IceBox.Service.IceStorm=IceStormService,32:createIceStorm

IceStorm.TopicManager.Endpoints=default -p 10000

IceStorm.Publish.Endpoints=default
IceStorm.Trace.TopicManager=1
IceStorm.Trace.Topic=2
IceStorm.Trace.Subscriber=1
IceStorm.Trace.SubscriberPool=1
IceStorm.Trace.Flush=1

#IceStorm.Send.Timeout=60000
IceStorm.Send.Timeout=1000
IceStorm.SubscriberPool.Size=1
IceStorm.SubscriberPool.SizeMax=5
IceStorm.SubscriberPool.SizeWarn=2
IceStorm.SubscriberPool.Timeout=1000

Ice.Override.ConnectTimeout=5000
Ice.Override.Timeout=5000

Ice.ThreadPool.Client.Size=3
Ice.ThreadPool.Client.SizeWarn=2
Ice.ThreadPool.Client.SizeMax=5

Ice.ThreadPool.Server.Size=3
Ice.ThreadPool.Server.SizeWarn=2
Ice.ThreadPool.Server.SizeMax=5

Freeze.DbEnv.IceStorm.DbHome=stormdb

Comments

  • benoit
    benoit Rennes, France
    Hi,

    Which QoS do your subscribers use to subscribe? If you're using oneway or batch oneway, the behavior you're seeing might be expected if the TCP/IP send buffer doesn't get filled up quickly enough before you plug back in the cable. That is, the send timeout kicks in only once sending oneway updates starts blocking (when the TCP/IP buffer is full).

    Did you try plugging out the cable for more than 10s? Also, it should work as you expect if you use the twoway or twoway ordered QoS.

    Cheers,
    Benoit.
  • Sorry, I should have mentioned that I'm using twoway QoS.
    I'm sure I set this up right because I can see it in the IceStorm trace messages.
    So I still don't understand the behaviour.

    I tried unplugging the cable for about 30s, but no longer. Perhaps 60s is the magic number? I'll try that when I get a chance this afternoon and post the result.
  • OK, I had a chance to re-run this test. I did the following three times:
    1. start everything running,
    2. pull out the cable
    3. wait till something happens.

    IceStorm does eventually kick the dead subscriber: in the three tests it happened after 45sec, 56sec, and 26sec. In IceStorm's traces I see:
    "topic publish failed: ConnectionI.cpp:483: Ice::TimeoutException".

    I didn't mention it in the initial post, but in case it's important, this instance of IceStorm is being used for various other purposes. All other uses have very low data rates, and are not sending traffic to the computer that gets unplugged.


    Thanks again,

    Alex
  • benoit
    benoit Rennes, France
    Hi Alex,

    Ok, I think this is still expected. Twoway updates are sent with AMI calls to the subscribers by IceStorm. The granularity of timeouts for AMI calls depends on how often the connection monitor thread checks pending AMI calls for timeout (see 33.3.5 Timeouts in the Ice manual).

    By default, the connection monitor thread wakes up every 60s and checks if AMI calls timed out or not. This explains why it takes time for IceStorm to detect the timeout (you should also take into account the retry which might add some delay -- here since you've set Ice.Override.ConnectTimeout to 5000, the retry shouldn't take more than 5s though.)

    You could try to set the Ice.MonitorConnections property to 10s for example, the timeout should kick in more quickly.

    Cheers,
    Benoit.
  • Excellent, I think I understand what's going on now. Thanks for the help, and thanks for the improvements to IceStorm!