Archived

This forum has been archived. Please start a new discussion on GitHub.

Ice ThreadPool configuration ambiguity

It is not clear to me how the 'global' (Ice.ThreadPool) thread pool configurations relate to the adapter-specific thread pool configuration values.

Say I am deploying HA-IceStorm in grid. Part of the configuration state for my icebox instance hosting HA-IceStorm will have the following config state:

<property name="IceBox.InheritProperties" value="1"/>
<property name="Ice.ThreadPool.Client.Size" value="10"/>
<property name="Ice.ThreadPool.Client.SizeMax" value="300"/>
<property name="Ice.ThreadPool.Server.Size" value="10"/>
<property name="Ice.ThreadPool.Server.SizeMax" value="300"/>

Then for the HA-IceStorm service, I have the following thread-pool configuration state:

<property name="${service}.TopicManager.ThreadPool.Size" value="2"/>
<property name="${service}.TopicManager.ThreadPool.SizeMax" value="20"/>
<property name="${service}.TopicManager.ThreadPool.SizeWarn" value="15"/>
<property name="${service}.Publish.ThreadPool.Size" value="10"/>
<property name="${service}.Publish.ThreadPool.SizeMax" value="30"/>
<property name="${service}.Publish.ThreadPool.SizeWarn" value="20"/><property name="${service}.Node.ThreadPool.Size" value="6"/>

It is not clear how the Ice.ThreadPool* configurations affect the adapater-specific thread-pool configurations. I think that the adapter-specific thread-pool configurations would only be influenced by the Ice.ThreadPool.Server* configurations, but does this mean that the Ice.ThreadPool.Server.SizeMax must be greater than the sum of all of the adapter-specific ThreadPool.SizeMax values? How do the Ice.ThreadPool.Server* configurations affect the adapter-specific thread-pool configurations?

And non-default Ice.ThreadPool.Client.* configurations are used only in the case of AMI, which I think HA-IceStorm uses. Does HA-IceStorm use AMI? I think that the Ice.ThreadPool.Client.SizeMax value applies to all of the client thread pools encapsulated by each of the Communicators deployed in the IceBox instance, but again, I am not sure. In other words, I've read your documentation about how to deploy threads as determined by the number of publishers and subscribers, etc, but I'm unsure how to precisely configure client thread-pool number. The bottom line is that I don't the Ice middleware to introduce artificial scaling limitations, I would rather let the hardward load be our guide. The environment is also pretty secure, with client-authenticated SSL for all invocations, so I am not really concerned about DOS.

Thanks

Dirk

Comments

  • matthew
    matthew NL, Canada
    I think you misunderstand how the thread pools work. By default there are two threads pools. The server side thread pool, and the client side thread pool. These thread pools are configured by the "Ice.ThreadPool.Server.*" and "Ice.ThreadPool.Client.*" set of properties.

    The client side thread pool manages outgoing connections, and is also used for dispatching AMI callbacks. The server side thread pool manages incoming connections, and is used by any object adapters that don't have their own thread pool configured.
    dhogan wrote: »
    ....
    It is not clear how the Ice.ThreadPool* configurations affect the adapater-specific thread-pool configurations....

    They have no effect on the adapter specific thread pool configurations.
    And non-default Ice.ThreadPool.Client.* configurations are used only in the case of AMI, which I think HA-IceStorm uses.

    Note that this isn't really correct. If your application uses bidirectional communications, then the invocations are dispatched from the client side thread pool.
    Does HA-IceStorm use AMI?

    Yes
    I think that the Ice.ThreadPool.Client.SizeMax value applies to all of the client thread pools encapsulated by each of the Communicators deployed in the IceBox instance, but again, I am not sure.

    What you really need to understand is which communicator gets which property. Any communicator which is configured with the property Ice.ThreadPool.Client.SizeMax will be affected. If you are really using "IceBox.InheritProperties" then, as the documentation says, each service inherits the configuration properties of the IceBox server’s communicator. So therefore, each service will have Ice.ThreadPool.Client.SizeMax configured to 10, with the configuration in your post, assuming that you are not overriding that in the service configuration.
    In other words, I've read your documentation about how to deploy threads as determined by the number of publishers and subscribers, etc, but I'm unsure how to precisely configure client thread-pool number.

    For HA IceStorm you can leave the value at the default (that is a single thread in the client side thread pool). There is little to no value in permitting concurrent AMI callbacks, since each AMI callback never blocks, and is extremely short lived.

    In fact, HA IceStorm tries reasonably hard to ensure that its thread pools are configured to values which will avoid causing issues, so I would recommend not configuring them yourself at all.
  • Specifying that Ice.ThreadPool* configurations apply only to adapters lacking their own thread-pool configurations might be a candidate for the Ice documentation.
    Note that this isn't really correct. If your application uses bidirectional communications, then the invocations are dispatched from the client side thread pool.

    What does bidirectional communications mean? Is it an invocation against a two-way proxy - i.e. an invocation which will write to a socket, and then read a return value? So if my Ice servant invokes a method against a two-way proxy(one service calling another), then this invocation is dispatched from the client thread pool, not from the adapter thread-pool which is running the invocation? Or do you mean that the client-side thread pool is running the ServerSocket.accept(...) for invocations against a given Ice servant, an socket connection which is then dispatched to the servant in the adapter thread pool?

    Dirk
  • matthew
    matthew NL, Canada
    I'm referring to a bidirectional connections as discussed in http://www.zeroc.com/doc/Ice-3.3.0-IceTouch/manual/Connections.38.7.html
  • dhogan wrote: »
    Specifying that Ice.ThreadPool* configurations apply only to adapters lacking their own thread-pool configurations might be a candidate for the Ice documentation.

    I think we have this already covered:

    http://www.zeroc.com/doc/Ice-3.3.0-IceTouch/manual/PropRef.50.7.html
    Ice.ThreadPool.Client.Size
    Ice.ThreadPool.Server.Size

    Synopsis

    Ice.ThreadPool.Client.Size=num
    Ice.ThreadPool.Server.Size=num
    Description

    A communicator creates two thread pools: the client thread pool dispatches AMI callbacks and incoming requests on bidirectional connections, and the server thread pool dispatches requests to object adapters. num is the initial and also minimum number of threads in the thread pool. The default value is one for both properties.
    An object adapter can also be configured with its own thread pool. See the object adapter properties for more information.
    Multiple threads for the client thread pool are only required for nested AMI invocations, or to allow multiple AMI callbacks to be dispatched concurrently.

    And:

    http://www.zeroc.com/doc/Ice-3.3.0-IceTouch/manual/PropRef.50.4.html
    adapter.ThreadPool.Size

    Synopsis

    adapter.ThreadPool.Size=num
    Description

    A communicator creates a default server thread pool that dispatches requests to its object adapters. An object adapter can also be configured with its own thread pool. This is useful in avoiding deadlocks due to thread starvation by ensuring that a minimum number of threads is available for dispatching requests to certain Ice objects. Section 32.10 describes thread pools in greater detail.
    num is the initial and also minimum number of threads in the thread pool. The default value is zero, meaning that an object adapter by default uses the communi cator's server thread pool. See Ice.ThreadPool.Server.Size for more information.
  • I read all of that information. The piece of information which is missing is the relationship between the Ice.ThreadPool.* configurations and the adapter-specific thread-pool configurations - i.e. the former only applies in the absence of the latter.
  • The two are independent.

    If you have Ice.ThreadPool.Server.Size=5 and MyAdapter.ThreadPool.Size=10, then MyAdapter gets a separate thread pool with ten threads. The remaining adapters all share the default thread pool with five threads.

    The SizeMax values are also independent and apply per thread pool. So, MyAdapter.ThreadPool.SizeMax=20 does not affect anything except the thread pool for MyAdapter.

    Cheers,

    Michi.
  • Ahh - thank-you for that final, crucial piece of information: i.e. that ALL adapters within a communicator without a thread-pool specified, share the single thread-pool specified by Ice.ThreadPool.Server.* (I thought that each adapter would get its own thread-pool configured by the Ice.ThreadPool.Server.* values).

    Communication is a perilous thing - it works imperfectly at best.

    Dirk
  • I'll make sure that the documentation is clear in this regard.

    Cheers,

    Michi.
  • Ok, let me summarize. Shouldn't this statement:
    32.10.3 Adapter Thread Pools

    The default behavior of an object adapter is to share the thread pools of its communicator

    be changed to
    Thedefault behavior of an object adapter is to share the server thread pool of its communicator.

    As I understand by default object adapter is using server pool, but not both.

    My second doubt is about client thread pool. How this settings influence regular (synchronous) method invocations? I expect that not at all because invocation is processed in caller thread. Am I right? So configuring client thread pool is relevant only when concurrent AMI are to take place.

    I expect that this:
    A communicator creates two thread pools: the client thread pool dispatches AMI callbacks and incoming requests on bidirectional connections, and the server thread pool dispatches requests to object adapters. num is the initial and also minimum number of threads in the thread pool. The default value is one for both properties.
    An object adapter can also be configured with its own thread pool. See the object adapter properties for more information.
    Multiple threads for the client thread pool are only required for nested AMI invocations, or to allow multiple AMI callbacks to be dispatched concurrently.
    confirms my presumptions but I would expect this explanation also right in the "32.10.1 Introduction to Thread Pools" chapter.
  • MarcinK wrote: »
    Shouldn't this statement:

    be changed to
    Thedefault behavior of an object adapter is to share the server thread pool of its communicator.

    As I understand by default object adapter is using server pool, but not both.

    No. Incoming requests on a bidir connection are handled by the shared client-side thread pool, unless the adapter has a separate client-side thread pool.
    My second doubt is about client thread pool. How this settings influence regular (synchronous) method invocations? I expect that not at all because invocation is processed in caller thread. Am I right? So configuring client thread pool is relevant only when concurrent AMI are to take place.

    Not quite. It also matters for incoming requests on bidirectional connections.
    I expect that this:

    confirms my presumptions but I would expect this explanation also right in the "32.10.1 Introduction to Thread Pools" chapter.

    It looks correct to me. Why do you think that needs changing?

    Cheers,

    Michi.