Archived

This forum has been archived. Please start a new discussion on GitHub.

Mixing AMI/SMI in the same connection results Timeout exception sometimes.

Previous Environments:
- Ice 3.4.2
- multiple IceGrid node; each server has only 1 thread in its thread pool.
- server expose 1 adapter with several methods
- The client call to IceGrid using SMI only.

Now:
so far there was no problem. after modifying the client, so that it uses AMI for some methods, and the ACM is disabled since the client uses bi-directional. Then we got Timeout exceptions from the client side from time to time.
I first suspect that since the server side uses only 1 thread, maybe the server was busy for the previous RPC, so that sometimes the subsequent call would not finished before the timeout. However my guess was wrong.

The client-side developer experiments, so she categorized proxies; one for SMI and another for AMI. These proxies are created by Proxy::ice_connectionid("smi"), and Proxy::ice_connectionid("smi"), so that calls to AMI uses separated connection from the connection for SMI RPC.

After that, the Timeout exception is gone.

If the original timeout exception was caused by the small number of thread in the server-side, using different connections should not prevent the timeout exception raised. Right?

So I'm forced to conclude that we may have a problem in the client-side, not the server-side.

Is that a known issue or, am I missing something?

Comments

  • benoit
    benoit Rennes, France
    Hi,

    It's not clear to me why it works with 2 separate connections. Do you use serialization for the thread pool (by setting the thread pool Serialize property to 1)?

    It's possible you introduced a deadlock with your AMI callbacks. This deadlock "resolves itself" thanks to the timeout. You could try to disable the timeouts and wait to see if your client ends up hanging instead. If it does hang, you can attach to the client process and analyze the stacks of each of the thread to figure out how to resolve the deadlock (if you provide us the stack traces, we can also help you analyzing the cause of the hang).

    Note that AMI callbacks are called by threads from the Ice client thread pool. By default, this thread pool only has 1 thread. If this thread is busy calling an AMI callback and your callback code for some reasons takes time to complete (or hangs waiting to acquire a lock which is held by another thread), other two-way calls might end up timing out because there are no more threads available in the Ice client thread pool to read the responses.

    One solution is to increase the client thread pool size (Ice.ThreadPool.Client.Size=100 for example) or if your AMI callback code needs to acquire some locks, you should ensure that those locks aren't held for too long (for example, you should avoid calling a two-way synchronous call with the lock held if AMI callbacks need to acquire the lock... similarly you should avoid calling two-way synchronous calls from AMI callbacks).

    Cheers,
    Benoit.