Archived

This forum has been archived. Please start a new discussion on GitHub.

what will happen when ice report thread pool size warning?

Hi, your gurus, thanks for your help in my last post, here I have some problems, also concerned with ice runtime thread pool.

I'm developing a service dispatching system using ICE, I create a dispatching thread pool, every thread in that pool continuing read requests from a queue, and if no new request, it sleeps. If it gets new request, then it try to dispatch this request to a remote machine. The remote machine may take some time to finish this request, so , one thread will wait for that request finish. The requests is recieved from an ice remote object, so I think it uses ice server thread pool, but it will sleep until the response is got from the dispatch thread. So from my design, you guys can see that my thread pool is worked as the ice runtime thread pool working in PerThreadPerConnection. (I known that this is not a good design for big concurrency, I will change it to use ice thread pool).

Now, here comes the problem, from my design, you can see that the max concurrency of service dispatching is the number of the threads in my thread pool. So I decide to test a concurrency of 200. So I set my thread pool in size of 200, and first, I set ice client and server thread pool both to 100 ( because I known ice runtime thread pool is in lead-follower model, it can handle more than 100 requests when set it to 100), and i make 200 requests, but I got lot of server thread pool warning, it reached 100, the max size. Then I set it to 200,300,400, the warning decresed, but still occurred. I was puzzled, I don't known why it still report warnning, because I do have only 200 requests, and if i plus the other incoming traffic, which happens when dispatch thread got answer of a request, it has only 400, not exceed the max thread pool size, and because i'm not running in perthreadperconnection mode, why it still report warnning?

and what will happen if I got this kind of warnning? does it mean that some requests are throw away?

thanks for any help

regards,
Xiang.

Comments

  • marc
    marc Florida
    A few notes and questions:
    • Why not use AMD (async message dispatch) instead of so many threads?
    • Why do you increase the size of the client-side thread pool? As long as you don't use AMI, there is no benefit in having more then on thread in the client-side thread pool.
    • The number of threads from the thread pool that are in use is usually higher than the number of threads currently dispatching operations. Each thread from the thread pool first unmarshals a requests, calls servant locators (if any), then dispatches the requests, marhals the return values, and most importantly, sends back the response. This means that even if only 100 threads are currently dispatching (i.e., executing your code), more than 100 threads from the thread pool can be busy with other things before they are returned to the thread pool.
    • The warning is a warning only, not an error message. It is mostly useful to debug deadlocks. No requests are thrown away if you get this warning. The requests will simply be dispatched as soon as a thread becomes available again.

    By the way, can you please update your project description in your user profile? "Dispatching System" is a bit generic...
  • marc wrote: »
    A few notes and questions:
    • Why not use AMD (async message dispatch) instead of so many threads?
    • Why do you increase the size of the client-side thread pool? As long as you don't use AMI, there is no benefit in having more then on thread in the client-side thread pool.
    • The number of threads from the thread pool that are in use is usually higher than the number of threads currently dispatching operations. Each thread from the thread pool first unmarshals a requests, calls servant locators (if any), then dispatches the requests, marhals the return values, and most importantly, sends back the response. This means that even if only 100 threads are currently dispatching (i.e., executing your code), more than 100 threads from the thread pool can be busy with other things before they are returned to the thread pool.
    • The warning is a warning only, not an error message. It is mostly useful to debug deadlocks. No requests are thrown away if you get this warning. The requests will simply be dispatched as soon as a thread becomes available again.

    By the way, can you please update your project description in your user profile? "Dispatching System" is a bit generic...

    thanks for your reply, why we don't use AMD, is because we want the requet first goes into our queue, so that we can control something of the request, such as priority or other controls, but the servent which recieve the requests can be used in AMI, so I have to enlarge client thread pool. Well, anyway, I've find out that there are some heartbeat request while concurrent dispatching, so maybe these traffic makes the ice runtime report warning. But thanks anyway.

    So from your description, I think if I invoke a block call to a remote servent, and this call will block for a long time, say 10 seconds, then it will not release one of the server thread in ice runtime, until this blocking call finished, is my understanding right?

    regards,
    Xiang.
  • matthew
    matthew NL, Canada
    thanks for your reply, why we don't use AMD, is because we want the requet first goes into our queue, so that we can control something of the request, such as priority or other controls

    I don't understand. If you don't use AMD you basically have no control over the dispatching of the request. It arrives from the client, is queued first by the operating system, and then the Ice runtime has to wait for a thread to become available to process the request, and then finally the request must be processed. At this point you have no choice but to process the request and return the reply. During the entire time that the request is being processed no further processing of requests can be done by that thread. With AMD the first steps remain the same. What is different is that once the request is dispatched you may choose when to process the request (possibly over many different threads) and when to return the reply to the caller. This is far more flexible, but of course more complicated :)
    So from your description, I think if I invoke a block call to a remote servent, and this call will block for a long time, say 10 seconds, then it will not release one of the server thread in ice runtime, until this blocking call finished, is my understanding right?

    Yes, exactly. This is why AMD is a good idea if your request processing can be time consuming.
  • Well, thanks for your explaination, I read Ice manual again, for the AMD chapter, I think it's the right tech for me to reduce the cost of server thread, also I will use AMI, to reduce my client thread number.

    But one question again :-)
    What will happen if I never call ice_response nor ice_exception ? Will the client got Timeout Exception? or what if client has got Timeout exception, and then I call ice_response or ice_exception on that AMD callback object?
  • soloman817 wrote: »
    Well, thanks for your explaination, I read Ice manual again, for the AMD chapter, I think it's the right tech for me to reduce the cost of server thread, also I will use AMI, to reduce my client thread number.

    But one question again :-)
    What will happen if I never call ice_response nor ice_exception ? Will the client got Timeout Exception? or what if client has got Timeout exception, and then I call ice_response or ice_exception on that AMD callback object?

    If you never call ice_response or ice_exception, your client will never receive any response or exception, and your server will continue to grow in size (i.e., this is a memory/resource leak).

    If there is a timeout configured for the client, your requests will eventually time out, and the whole connection is closed (since timeouts are treated as fatal connection errors, see our user manual for details).

    Note that such client timeouts would not resolve any resource leaks in the server caused by incomplete AMD calls. You must make sure that your code calls either ice_response or ice_exception for every AMD call, otherwise your program will leak.
  • Thanks again!

    So if I never call ice_response nor ice_exception, the client can only get fatal exception. So what I mean is when I use exit(0) to quit my server, can client quit as well? So I guess it will get Ice::ConnectionLostException, and this is also a fatal exception. Only problem is if my processing thread crashed, say report exception to Ice::Application or Ice::Service, but the server is still running, then this could hangs up client and may cause memory leak.

    Well, I have re-design my program, I decide to use lead/followers mode to make a dispatch thread pool, my program is act as a service dispatch gateway, it managed clients and service providers which connected to my dispatch gateway, try to dispatch a client's request to a connected service provider, so I have to implement a dispatch thread of my own, which use AMD to queue incoming clients' request, and the leader/followers thread pool try to find an avaiable service and then use AMI to dispatch it. The way I use AMI to do dispatching could reduce the operation time of leader thread.

    But another question, the leader/followers mode need to listen on a set of handles, which could be thought of a set of monitors, but I could not find any way in Ice runtime, which could wait on more than one monitor, like Windows API waitForMutilObject(...), is there any suggestion on how to wait on a set of monitors? and when wake up, could the thread find for which monitor it is waked? Means to known which event awake it?

    regards,
    xiang.
  • Can you please update your project in your user profile? We do not provide free support if the project name or description is missing. ("Dispatching System" is not a project name or description, every Ice server is a dispatching system.)
  • marc wrote: »
    Can you please update your project in your user profile? We do not provide free support if the project name or description is missing. ("Dispatching System" is not a project name or description, every Ice server is a dispatching system.)

    Well, sorry, I just don't known how to name the system, it's act like a service forward gateway, many server connect to it, register their service, and client send requests, and this program try to forward requests, that's it, so now I named it 'Dynamic Service Forward Gateway', is that right?