Archived

This forum has been archived. Please start a new discussion on GitHub.

icebox runs slowly

Hi, all! I am using icebox as my pub/sub service refer to sample code 'Ice-3.4.1\cpp\demo\IceStorm\clock'. sometimes when i pub a new message, the subscriber receives this message about 20 seconds later or more. anyone knows why there is so great latency? Did i config incorrectly? thanks!

Comments

  • buptstehc wrote: »
    Hi, all! I am using icebox as my pub/sub service refer to sample code 'Ice-3.4.1\cpp\demo\IceStorm\clock'. sometimes when i pub a new message, the subscriber receives this message about 20 seconds later or more. anyone knows why there is so great latency? Did i config incorrectly? thanks!

    or is there any way to show when icebox receives message from publisher and when to send to subscriber?
  • benoit
    benoit Rennes, France
    Hi,

    Note that we only provide free support on the forums for the latest Ice versions so I recommend to upgrade to Ice 3.5.1.

    You can enable protocol tracing to see messages sent and received by IceBox. You need to set Ice.Trace.Protocol=1 in the IceStorm service configuration file to enable protocol tracing. Unless you use batch oneway subscribers, IceStorm should forward the messages to the subscribers as soon as it receives them. If you use batch oneway subscribers, IceStorm flushes the batch messages every N seconds where N is specified with the <service-name>.Flush.Timeout property (default is every seconds).

    Cheers,
    Benoit.
  • benoit wrote: »
    Hi,

    Note that we only provide free support on the forums for the latest Ice versions so I recommend to upgrade to Ice 3.5.1.

    You can enable protocol tracing to see messages sent and received by IceBox. You need to set Ice.Trace.Protocol=1 in the IceStorm service configuration file to enable protocol tracing. Unless you use batch oneway subscribers, IceStorm should forward the messages to the subscribers as soon as it receives them. If you use batch oneway subscribers, IceStorm flushes the batch messages every N seconds where N is specified with the <service-name>.Flush.Timeout property (default is every seconds).

    Cheers,
    Benoit.

    Thanks for your reply. since icebox is running in the prod environment, i will upgrade it later. the subscribers run in the oneway mode, not the batch one. here is the code snippet in c#:
                IceStorm.TopicManagerPrx manager = IceStorm.TopicManagerPrxHelper.checkedCast(
                ic.propertyToProxy("TopicManager.Proxy"));
                if (manager == null) {
                    Console.Error.WriteLine("invalid proxy");
                    return;
                }
    
                //
                // Retrieve the topic.
                //
                string topicName = "ps1";
                IceStorm.TopicPrx topic = null;
                try {
                    topic = manager.retrieve(topicName);
                }
                catch (IceStorm.NoSuchTopic) {
                    Console.Error.WriteLine("No topic found!");
                    return;
                }
    
                Ice.ObjectAdapter adapter = ic.createObjectAdapter("Jupiter.Subscriber");
                Ice.Identity subId = new Ice.Identity(Guid.NewGuid().ToString(), "");
                Ice.ObjectPrx subscriber = adapter.add(new PsI(p), subId).ice_oneway();
    
                Dictionary<string, string> qos = new Dictionary<string, string>();
                //qos["retryCount"] = "-1";
                try {
                    topic.subscribeAndGetPublisher(qos, subscriber);
                }
                catch (IceStorm.AlreadySubscribed) {
                    Console.Error.WriteLine("reactivating persistent subscriber");
                }
    
                adapter.activate();
                ic.waitForShutdown();
                topic.unsubscribe(subscriber);
    

    and this is the configuration file for icestorm:
    #
    # The IceStorm service instance name.
    #
    IceStorm.InstanceName=JupiterIceStorm
    
    #
    # This property defines the endpoints on which the IceStorm
    # TopicManager listens.
    #
    IceStorm.TopicManager.Endpoints=default -p 10000
    
    #
    # This property defines the endpoints on which the topic
    # publisher objects listen. If you want to federate
    # IceStorm instances this must run on a fixed port (or use
    # IceGrid).
    #
    IceStorm.Publish.Endpoints=tcp -p 10001:udp -p 10001
    
    #
    # IceStorm SQL configuration if using SQL database.
    #
    #Ice.Plugin.DB=IceStormSqlDB:createSqlDB
    #IceStorm.SQL.DatabaseType=QSQLITE
    #IceStorm.SQL.DatabaseName=db/IceStorm.db
    
    #
    # TopicManager Tracing
    #
    # 0 = no tracing
    # 1 = trace topic creation, subscription, unsubscription
    # 2 = like 1, but with more detailed subscription information
    #
    IceStorm.Trace.TopicManager=2
    
    #
    # Topic Tracing
    #
    # 0 = no tracing
    # 1 = trace unsubscription diagnostics
    #
    IceStorm.Trace.Topic=1
    
    #
    # Subscriber Tracing
    #
    # 0 = no tracing
    # 1 = subscriber diagnostics (subscription, unsubscription, event
    #     propagation failures)
    #
    IceStorm.Trace.Subscriber=1
    
    #
    # Amount of time in milliseconds between flushes for batch mode
    # transfer. The minimum allowable value is 100ms.
    #
    IceStorm.Flush.Timeout=2000
    
    #
    # Network Tracing
    #
    # 0 = no network tracing
    # 1 = trace connection establishment and closure
    # 2 = like 1, but more detailed
    # 3 = like 2, but also trace data transfer
    #
    #Ice.Trace.Network=1
    
    #
    # This property defines the home directory of the Freeze 
    # database environment for the IceStorm service.
    #
    Freeze.DbEnv.IceStorm.DbHome=db
    
    IceStorm.Send.Timeout=600
    IceStorm.Discard.Interval=600
    

    and this is for icebox:
    #
    # The IceBox server endpoint configuration. This endpoint is only used
    # to communicate with the IceBox ServiceManager object (such as when
    # using iceboxadmin to shutdown the server).
    #
    # The IceStorm service has its own endpoints (see config.service).
    #
    IceBox.ServiceManager.Endpoints=tcp -p 9998
    
    #
    # The IceStorm service. The service is configured using a separate
    # configuration file (see config.service).
    #
    IceBox.Service.IceStorm=IceStormService,34:createIceStorm --Ice.Config=config.service
    
    #
    # Warn about connection exceptions
    #
    #Ice.Warn.Connections=1
    
    #
    # Network Tracing
    #
    # 0 = no network tracing
    # 1 = trace connection establishment and closure
    # 2 = like 1, but more detailed
    # 3 = like 2, but also trace data transfer
    #
    #Ice.Trace.Network=1
    
    #
    # Protocol Tracing
    #
    # 0 = no protocol tracing
    # 1 = trace protocol messages
    #
    Ice.Trace.Protocol=1
    

    By the way, since the tracing for ice logs in the console, how can i redirect to file? thanks!
  • benoit
    benoit Rennes, France
    Hi,

    You can redirect the logging to a file by setting the Ice.LogFile property in the IceStorm service configuration file.

    Note that this will generate lots of traces if the IceStorm service receives and forwards many events in your production environment. I would recommend to try to reproduce the problem in a test environment instead.

    Is the problem easily reproducible? What does the subscriber do when it receives an event? Such a delay could be expected if for example the subscriber performs a task that takes 20s upon receiving an event and only has a single thread to dispatch events received from IceStorm.

    Cheers,
    Benoit.
  • benoit wrote: »
    Hi,

    You can redirect the logging to a file by setting the Ice.LogFile property in the IceStorm service configuration file.

    Note that this will generate lots of traces if the IceStorm service receives and forwards many events in your production environment. I would recommend to try to reproduce the problem in a test environment instead.

    Is the problem easily reproducible? What does the subscriber do when it receives an event? Such a delay could be expected if for example the subscriber performs a task that takes 20s upon receiving an event and only has a single thread to dispatch events received from IceStorm.

    Cheers,
    Benoit.

    Thanks,Benoit. During these days, this latency problem occurs again, which will kills me. I found a wield phenomenon: suppose there are two messages A and B, which both have id and timestamp fileds. the publisher pubs A every 5 seconds and B at random time. by comparing the timestamp when subscriber receives A, all A will delayed about 10 minutes, which means if the first A is sent at time N, the subscriber receives at N + 10m, N + 10m + 5s,N + 10m + 5s + 5s,... and B is delayed in this interval.
    since the subscriber runs in a single thread, which will do this work:
    1.receives message
    2.insert into db
    3.receives message again.
    according to A's log, during these time, the subscriber won't be suspended during inserting data to db since it will receives message although all messages be delayed 10m. So why B is also delayed? my another question is whether icestorm will cache message and deliver at later, for example, if the connection is lost with subscriber.however, ice should do reconnect work automatically, the interval between the lost could't be 10m or more. please help!
    i have another question about icestorm. whether icestorm will hold a queue for every subscriber, if subscriber processed message slowly, and there are plenty of A in the queue before B, and A is delivered every 5 seconds, so B is delayed 10m?
  • benoit
    benoit Rennes, France
    Hi,

    It's not clear when the B events are received by the subscriber. In theory, they should be received in the same order as they were published so it would be logical if the B events were dispatched with the same 10 minute delay if the A events are received with such a delay as well.

    Without more information it's difficult to say why you observe such a delay. The first thing you need to check is whether or not the subscriber is capable of processing the events fast enough. If the subscriber is too slow, it will inevitably end-up receiving events with a delay which will increase over time (and the event queue for the subscriber in IceStorm will also end-up growing causing IceStorm to consume more memory). So if your publisher publishes A events each 5 seconds, you must ensure that it takes much less than 5 seconds for your subscriber to process those events.

    If IceStorm encounters a connection failure and can't re-establish the connection immediately, it will clear the event queue for the subscriber and re-try establishing the connection upon receiving the next event. How many times IceStorm will retry establishing the connection to the subscriber depends on retry count quality of service configured for the subscriber (see here for more information).

    Cheers,
    Benoit.