running out of threadpool limit

LAstakhov

Here is a problem:

We have one dedicated server and 60 clients. The server is configured as following:
Ice.ThreadPool.Server.Size=64
Ice.ThreadPool.Client.Size=32

It works fine until it comes with a warning of thread pool size and falling into a deadlock after the final warning to the log:

thread pool `Ice.ThreadPool.Server' is running low on threads
Size=64, SizeMax=64, SizeWarn=51

It looks like there is no any free thread available anymore to send a response. So every locked thread is waiting for at least one free thread to be available but it will never happen.

What would be your advise of how to locate this kind of problems in a code? Maybe Ice has some configuration properties for deadlock diagnostic?


Best regards, Leo

benoit

Hi Leo,

The best way to figure out the cause of a deadlock is to attach to the process with the debugger and take a look at the stack trace of each thread. This will tell you why the threads are blocked (waiting on a mutex? waiting on twoway request to return? etc).

I also recommend reading Bernard's newsletter articles about deadlocks here. It should give you a better idea on the possible reasons for deadlocks and how to avoid them.

Note however that the articles are a bit out of date with respect to AMI. Since Ice 3.3.0, AMI calls are guaranteed to not block. If you don't want to block a thread from the Ice server thread pool by making a potentially blocking Ice remote call, you can now use AMI.

Cheers,
Benoit.