Archived

This forum has been archived. Please start a new discussion on GitHub.

IceStorm: Publishing from a Servant

Hello,

I am currently using Glacier2 and IceStorm in my project (Ice 3.5.0). I am using IceStorm to set up a publisher-subscriber system with several topics.

I have a situation where when a subscriber on topic A receives a message, the Servant for that subscriber tries to publish a message (on a different topic, call it topic B). I was previously only using IceStorm (without Glacier2) and doing this never seemed to be a problem. After adding the Glacier2 functionality, however, I am having a problem where trying to publish a message in the Servant will cause it to hang permanently (specifically, on the line where I call a method on the proxy). If I instead run the publishing code in a separate thread, the problem does not occur.

Another strange behavior I am encountering: if I publish a message on topic B outside of the servant for the subscriber to topic A, then that servant will henceforth be able to publish to topic B without encountering the hanging behavior.

I am puzzled by all this, and was hoping someone had some insight as to why this is happening. Is it a bad idea, in general, to try to publish in a subscriber callback? Why would it work in some cases, but hang in others?

Comments

  • bernard
    bernard Jupiter, FL
    Hello Axel,

    Welcome to our forums!

    This hang occurs because you're running out of threads in your client thread pool when you use Glacier2.

    Without Glacier2, the message from IceStorm is dispatched in your subscriber by a thread from its server thread pool. When this servant then sends a two-way synchronous request, the reply to this request is processed by a thread of the subscriber's client thread pool. This works well with the default thread pool sizes - a single thread in each thread pool. See The Ice Threading Model - Ice 3.5 - ZeroC for more details.

    With Glacier, the connection between the subscriber and the Glacier2 router is bidirectional, and as the result the message from IceStorm is dispatched in your subscriber by a thread from its client thread pool. When this servant then sends a two-way synchronous request, with the default configuration (1 thread in the client thread pool), there is no thread is the client thread pool to process the reply and you get a hang. We warn about this issue in the Ice manual: Callbacks through Glacier2 - Ice 3.5 - ZeroC.

    In your situation, you could increase the number of threads in your subscriber's client thread pool. Alternatively, you could use AMI for the nested request, to release the one thread in your client thread pool and allow this thread to process the reply to this request. With IceStorm, this reply is always "void" or a system exception.

    I hope it's clearer now!

    Best regards,
    Bernard
  • Hi Bernard,

    Thank you for your quick and insightful reply. I suspected something like that was going on, but couldn't quite get a handle on it.

    Your response makes sense to me, and as you predicted, increasing the number of threads in the subscriber client thread pool solves the problem. Although I admit I am still unsure why I observed that publishing one message outside of the servant would subsequently solve the blocking problem (maybe publishing the first message has the side-effect of increasing the number of threads in the subscriber client thread pool?).

    Thank you again,

    Axel
  • bernard
    bernard Jupiter, FL
    Hi Axel,

    The only way to increase the number of threads in a thread pool is through configuration.
    Another strange behavior I am encountering: if I publish a message on topic B outside of the servant for the subscriber to topic A, then that servant will henceforth be able to publish to topic B without encountering the hanging behavior.

    The location of the call - within the servant or outside this servant's code - does not matter. What matters is the thread making the call. If this is call made from the thread that dispatches the message from IceStorm, then you will need an extra thread in your client thread pool to make a nested two-way synchronous call (when using Glacier2 or bidirectional connections).

    Best regards,
    Bernard
  • Hi Bernard,
    bernard wrote: »
    The only way to increase the number of threads in a thread pool is through configuration.


    The location of the call - within the servant or outside this servant's code - does not matter. What matters is the thread making the call. If this is call made from the thread that dispatches the message from IceStorm, then you will need an extra thread in your client thread pool to make a nested two-way synchronous call (when using Glacier2 or bidirectional connections).

    Sorry, I was not clear when I talked about who was making the call. What I meant was the following two steps:

    1) Some non-subscriber thread (e.g. the application's main thread) publishes to topic B.

    2) Then, the subscriber receives a message from some other topic A. As a part of the callback, it publishes to topic B (executed in the subscriber dispatch thread).

    If I do 1) first, then mysteriously, 2) will never hang.
    If I do only 2), then it will hang, as predicted by your previous explanation.

    I was curious as to why this happens. But if it's not immediately obvious, I'm happy to just leave it be, as my main problem is solved!

    Best,

    Axel
  • benoit
    benoit Rennes, France
    Hi Axel,

    When you get a hang, the best is to attach with the debugger and get a dump of all the threads and see where they are blocking, this will often provide the explanation for the hang. Thad said, I have a possible explanation for your scenario where it didn't hang when you first published from a separate thread on the topic.

    When using Glacier2, an invocation on a proxy from the client (the topic publisher in your case) will under the hood cause another invocation (addProxies) on the Glacier2 Ice::Router interface to add the proxy to the Glacier2 routing table. This operation is done only once for each proxy -- subsequent calls on the proxy are then sent directly from the client to the server through Glacier2.

    In your scenario, when you publish on topic B from a separate thread in 1), the Ice::Router:: addProxies call is made from this thread. The response from the addProxies call is received by the client thread from the client thread pool and your publish call is then sent to IceStorm through Glacier2. When 2) invokes again on the publisher for topic B, there's no need to call Ice::Router:: addProxies again since it was already done.

    When you skip step 1), the Ice::Router::addProxies call is made from the client thread pool thread and this is causing a hang because there are no more threads in the client thread pool to read the response for this call (as explained by Bernard in previous posts).

    You could easily verify this by enabling protocol tracing on your client with the Ice.Trace.Protocol property set to 2. At the time of the hang, you should see a call to addProxies.

    Increasing the number of threads in the client thread pool is one solution. Another solution would be to use AMI to send the publisher requests asynchronously. AMI calls are guaranteed to not block (add the addProxies call which is made under the hood will also be performed asynchronously).

    Let us know if you need more information on this!

    Cheers,
    Benoit.
  • Hi Benoit,
    benoit wrote: »
    When you skip step 1), the Ice::Router::addProxies call is made from the client thread pool thread and this is causing a hang because there are no more threads in the client thread pool to read the response for this call (as explained by Bernard in previous posts).

    You could easily verify this by enabling protocol tracing on your client with the Ice.Trace.Protocol property set to 2. At the time of the hang, you should see a call to addProxies.

    You are completely right. The trace showed this exact situation. Thanks for taking the time to explain this to me and satisfy my curiosity! I am using the thread pool size increase solution for the moment, but I will definitely look into AMI as well.

    Merci beaucoup Benoit et Bernard!

    - Axel