IceGrid Performance

ramayana · June 2012

Hi again!

First i have changed Hello World app so server calculate sqrt and returning result to client. I set Ice.ThreadPool.Server.SizeMax to 10 on server and wrote for loop (i.e. from 1 to 100000), which is calling my sqrt function. I measure the time from start to end a loop.

Then i have changed IceGrid/replication demo that it did the same - calculate sqrt. But when i craeted the same for loop IceGrid app needs almost twice as much time. But in this demo are 4 serwers, so why it is much slower? I added to config.master

IceGrid.Registry.Client.ThreadPool.Size=1
IceGrid.Registry.Client.ThreadPool.SizeMax=10
IceGrid.Registry.Internal.ThreadPool.Size=1
IceGrid.Registry.Internal.ThreadPool.SizeMax=10

and i deleted in every config file
Ice.StdErr=db/node1/stderr.txt
Ice.StdOut=db/node1/stdout.txt

and every tracing, but it did not help... Why 4 serwers (with 10 threads for each of them) are slower than one (with 10 threads)?

Thanks for help.

EDIT: have forgotten to wrote that on the beggining i did not change config files, like adding ThreadPool parameters or deleting tracing.

benoit · June 2012

Hi,

Why did you modify the IceGrid thread pool configuration? You shouldn't have to modify the IceGrid thread pool configuration, it should work with the default configuration.

It isn't clear to me how exactly is written your test and what kind of load balancing you are using. You would need to detail this, eventually providing your test case.

In any case, few comments:

Using 10 threads in each server is useful only if you run each server on a separate machine and if each machine has at least 10 cores or if the implementation of your servants perform blocking operations (IO, database operations, etc). If the implementation of your servants isn't blocking (e.g.: sqrt implementation), having 10 threads running concurrently won't be any faster than having 1 thread (it will in fact be worse as the cost of thread context switches will be higher...).
If your servant method implementation take very little time to complete (a single sqrt computation should be quite quick to complete...). The latency between the client and the server is likely to be the dominant factor in your performance test... Even if Ice is very fast, a remote Ice call will always be slower than a direct programming function call.
If your client needs to query the IceGrid registry before each call to one of your server for load balancing purposes, the latency will be even higher. Therefore, this type of load balancing if interesting only for long running server operations.

So if your servers just do a single "sqrt" computation and then return and if you use IceGrid per-request load balancing in your client, I'm not surprised the performances are worst when using 4 servers. In this case, using a single server (with a number of threads equal to the number of cores) and no per-request load balancing will be faster.

On the opposite, if your server implementation makes long running computations for each Ice invocation (e.g.: one Ice invocation results in 100000 sqrt calls), if you have 4 servers running on different machines where each server is configured with a number of threads equal to the number of cores of the machine and if your client uses per-request load balancing... then I would expect such a scenario to perform better with 4 servers than a single server.

Cheers,
Benoit.

ramayana · June 2012

Hi Benoit,
Actually i am exaggerating with threads, but for my defense i tell you that i did that later.

So, i don't understand exactly what is load balancing. I cut delay option from Client.cpp and i thought that everything is alright.

I attached project. Meybe it's okay, but i don't understand IceGrid...

FILE

Now i will try run two nodes on each machine, will see how it works then

benoit · June 2012

Hi,

Could you detail what you don't understand and what performance numbers you are looking for?

IceGrid load balancing allows to spread the load of Ice invocations on different servers. As explained above, increasing the number of threads or the number of servers doesn't automatically improves the performances.

If I understand your test case, you run a client which calls N times on one of the load balanced servers. The server servant implementation just does a simple sqrt computation. What do you expect to measure with this test?

As explained earlier, the load balancing has a small cost as it implies making an additional invocation on the IceGrid locator for each invocation (with the configuration used by your test). In general this cost is insignificant compared to the time taken by the call to complete.

Cheers,
Benoit.

ramayana · June 2012

The most important questions is if the calls from client actually goes to nodes equally? If i am understand correctly if i run 2 nodes on A machine and 2 nodes on B machine time for calling sqrt function 100000 times will be better than simple client -> server. And why i need replica? It increases numbers of nodes?
And one more questions. If i run element A on different machine that element B both has to listening opposite ip? So A has to listen B ip, and B has to listen A ip?

Thanks for your help

benoit · June 2012

Hi,

If you use round-robin for the load balancing, the calls will be equally spread on all the replicas. I believe I already answered your question regarding replication

.

As for your last question, in general if an Ice server needs to dispatch requests from clients, it needs to listen on a IP address and port. If both A and B expose Ice objects with servants, both will need to listen on an IP address and port.

There is also the possibility of using bi-directional connections so that only one process listens on an IP/port but this only works in some specific use cases where an Ice client always establishes a connection the Ice server first (and the server can call back the Ice client through this same Ice connection). For more information I recommend going through the Ice manual and the Ice demos.

Cheers,
Benoit.

ramayana · June 2012

You are my savior. And, i hope, last questions. i have client on other machine than other files, master, nodes and replica are on the same machine and i modified config.client on client machine:

ce.Default.Locator=DemoIceGrid/Locator:default -h 192.168.128.110 -p 12000:default -h 192.168.128.110 -p 12001:default -h 192.168.128.110  -p 12002

someIP is IP of server machine, do i nedd to modify config files of master, replicas and nodes? In situation when i don't modified them and in other situation when i do (adding -h someIP in Default.Locator and IceGrid registry configuration in every file) when i run client i get:

-- 06/14/12 11:35:57.036 ./client: Locator: searching for object by id
   object = hello
-- 06/14/12 11:35:57.037 ./client: Network: trying to establish tcp connection to 192.168.128.110:12000
-- 06/14/12 11:35:57.038 ./client: Network: tcp connection established
   local address = 192.168.128.149:36409
   remote address = 192.168.128.110:12000
-- 06/14/12 11:35:57.071 ./client: Locator: searching for adapter by id
   adapter = ReplicatedHelloAdapter
-- 06/14/12 11:35:57.078 ./client: Locator: no endpoints configured for adapter
   adapter = ReplicatedHelloAdapter
-- 06/14/12 11:35:57.078 ./client: Locator: searching for object by id
   object = hello
-- 06/14/12 11:35:57.079 ./client: Locator: searching for adapter by id
   adapter = ReplicatedHelloAdapter
-- 06/14/12 11:35:57.121 ./client: Locator: no endpoints configured for adapter
   adapter = ReplicatedHelloAdapter
!! 06/14/12 11:35:57.121 ./client: error: Reference.cpp:1409: Ice::NoEndpointException:
   no suitable endpoint available for proxy `hello -t'
-- 06/14/12 11:35:57.121 ./client: Network: closing tcp connection
   local address = 192.168.128.149:36409
   remote address = 192.168.128.110:12000

ramayana · June 2012

It's quite strange but everything is working correct. I have started from the beggining and now it's working! Now i will make some test.

benoit, thanks again for help!

Archived

IceGrid Performance

Comments

Categories