Home Help Center

IceGrid Performance

ramayanaramayana Member Stanislaw ArtymowskiOrganization: Instytut Monitorowania MediówProject: Testing and learning ICE
Hi again!

First i have changed Hello World app so server calculate sqrt and returning result to client. I set Ice.ThreadPool.Server.SizeMax to 10 on server and wrote for loop (i.e. from 1 to 100000), which is calling my sqrt function. I measure the time from start to end a loop.

Then i have changed IceGrid/replication demo that it did the same - calculate sqrt. But when i craeted the same for loop IceGrid app needs almost twice as much time. But in this demo are 4 serwers, so why it is much slower? I added to config.master
IceGrid.Registry.Client.ThreadPool.Size=1
IceGrid.Registry.Client.ThreadPool.SizeMax=10
IceGrid.Registry.Internal.ThreadPool.Size=1
IceGrid.Registry.Internal.ThreadPool.SizeMax=10

and i deleted in every config file
Ice.StdErr=db/node1/stderr.txt
Ice.StdOut=db/node1/stdout.txt

and every tracing, but it did not help... Why 4 serwers (with 10 threads for each of them) are slower than one (with 10 threads)?

Thanks for help.


EDIT: have forgotten to wrote that on the beggining i did not change config files, like adding ThreadPool parameters or deleting tracing.

Comments

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Why did you modify the IceGrid thread pool configuration? You shouldn't have to modify the IceGrid thread pool configuration, it should work with the default configuration.

    It isn't clear to me how exactly is written your test and what kind of load balancing you are using. You would need to detail this, eventually providing your test case.

    In any case, few comments:
    • Using 10 threads in each server is useful only if you run each server on a separate machine and if each machine has at least 10 cores or if the implementation of your servants perform blocking operations (IO, database operations, etc). If the implementation of your servants isn't blocking (e.g.: sqrt implementation), having 10 threads running concurrently won't be any faster than having 1 thread (it will in fact be worse as the cost of thread context switches will be higher...).
    • If your servant method implementation take very little time to complete (a single sqrt computation should be quite quick to complete...). The latency between the client and the server is likely to be the dominant factor in your performance test... Even if Ice is very fast, a remote Ice call will always be slower than a direct programming function call.
    • If your client needs to query the IceGrid registry before each call to one of your server for load balancing purposes, the latency will be even higher. Therefore, this type of load balancing if interesting only for long running server operations.

    So if your servers just do a single "sqrt" computation and then return and if you use IceGrid per-request load balancing in your client, I'm not surprised the performances are worst when using 4 servers. In this case, using a single server (with a number of threads equal to the number of cores) and no per-request load balancing will be faster.

    On the opposite, if your server implementation makes long running computations for each Ice invocation (e.g.: one Ice invocation results in 100000 sqrt calls), if you have 4 servers running on different machines where each server is configured with a number of threads equal to the number of cores of the machine and if your client uses per-request load balancing... then I would expect such a scenario to perform better with 4 servers than a single server.

    Cheers,
    Benoit.
  • ramayanaramayana Member Stanislaw ArtymowskiOrganization: Instytut Monitorowania MediówProject: Testing and learning ICE
    Hi Benoit,
    Actually i am exaggerating with threads, but for my defense i tell you that i did that later.

    So, i don't understand exactly what is load balancing. I cut delay option from Client.cpp and i thought that everything is alright.

    I attached project. Meybe it's okay, but i don't understand IceGrid...

    FILE

    Now i will try run two nodes on each machine, will see how it works then
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Could you detail what you don't understand and what performance numbers you are looking for?

    IceGrid load balancing allows to spread the load of Ice invocations on different servers. As explained above, increasing the number of threads or the number of servers doesn't automatically improves the performances.

    If I understand your test case, you run a client which calls N times on one of the load balanced servers. The server servant implementation just does a simple sqrt computation. What do you expect to measure with this test?

    As explained earlier, the load balancing has a small cost as it implies making an additional invocation on the IceGrid locator for each invocation (with the configuration used by your test). In general this cost is insignificant compared to the time taken by the call to complete.

    Cheers,
    Benoit.
  • ramayanaramayana Member Stanislaw ArtymowskiOrganization: Instytut Monitorowania MediówProject: Testing and learning ICE
    The most important questions is if the calls from client actually goes to nodes equally? If i am understand correctly if i run 2 nodes on A machine and 2 nodes on B machine time for calling sqrt function 100000 times will be better than simple client -> server. And why i need replica? It increases numbers of nodes?
    And one more questions. If i run element A on different machine that element B both has to listening opposite ip? So A has to listen B ip, and B has to listen A ip?

    Thanks for your help
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    If you use round-robin for the load balancing, the calls will be equally spread on all the replicas. I believe I already answered your question regarding replication ;).

    As for your last question, in general if an Ice server needs to dispatch requests from clients, it needs to listen on a IP address and port. If both A and B expose Ice objects with servants, both will need to listen on an IP address and port.

    There is also the possibility of using bi-directional connections so that only one process listens on an IP/port but this only works in some specific use cases where an Ice client always establishes a connection the Ice server first (and the server can call back the Ice client through this same Ice connection). For more information I recommend going through the Ice manual and the Ice demos.

    Cheers,
    Benoit.
  • ramayanaramayana Member Stanislaw ArtymowskiOrganization: Instytut Monitorowania MediówProject: Testing and learning ICE
    You are my savior. And, i hope, last questions. i have client on other machine than other files, master, nodes and replica are on the same machine and i modified config.client on client machine:
    ce.Default.Locator=DemoIceGrid/Locator:default -h 192.168.128.110 -p 12000:default -h 192.168.128.110 -p 12001:default -h 192.168.128.110  -p 12002
    

    someIP is IP of server machine, do i nedd to modify config files of master, replicas and nodes? In situation when i don't modified them and in other situation when i do (adding -h someIP in Default.Locator and IceGrid registry configuration in every file) when i run client i get:
    -- 06/14/12 11:35:57.036 ./client: Locator: searching for object by id
       object = hello
    -- 06/14/12 11:35:57.037 ./client: Network: trying to establish tcp connection to 192.168.128.110:12000
    -- 06/14/12 11:35:57.038 ./client: Network: tcp connection established
       local address = 192.168.128.149:36409
       remote address = 192.168.128.110:12000
    -- 06/14/12 11:35:57.071 ./client: Locator: searching for adapter by id
       adapter = ReplicatedHelloAdapter
    -- 06/14/12 11:35:57.078 ./client: Locator: no endpoints configured for adapter
       adapter = ReplicatedHelloAdapter
    -- 06/14/12 11:35:57.078 ./client: Locator: searching for object by id
       object = hello
    -- 06/14/12 11:35:57.079 ./client: Locator: searching for adapter by id
       adapter = ReplicatedHelloAdapter
    -- 06/14/12 11:35:57.121 ./client: Locator: no endpoints configured for adapter
       adapter = ReplicatedHelloAdapter
    !! 06/14/12 11:35:57.121 ./client: error: Reference.cpp:1409: Ice::NoEndpointException:
       no suitable endpoint available for proxy `hello -t'
    -- 06/14/12 11:35:57.121 ./client: Network: closing tcp connection
       local address = 192.168.128.149:36409
       remote address = 192.168.128.110:12000
    
    
  • ramayanaramayana Member Stanislaw ArtymowskiOrganization: Instytut Monitorowania MediówProject: Testing and learning ICE
    It's quite strange but everything is working correct. I have started from the beggining and now it's working! Now i will make some test.

    benoit, thanks again for help!
Sign In or Register to comment.