Archived

This forum has been archived. Please start a new discussion on GitHub.

IceGrid usage - Server Template Initantiation

Hi,

I am using Ice 3.2.1 on freebsd 7.

I have two interfaces.
module DemoList
{
	interface ListMaster
	{
	    void createList(string name);
	};

	interface ListManager
	{
	    void add(string key, string value);
	    void delete(string key);
	    void deleteList();
	};
};

Using ListMaster interface, I do create and initialise Berkeley DB hash table, with unique list name.

I am using multiple machines and IceGrid to replicate ListMaster interface using round robin policy under ListMasterApp application.

So, one set of clients create these lists on different servers, in round robin fashion on available node machines.

Other set of clients need to connect back to these lists on different nodes using ListManager interface. To keep the available nodes transparent, we are using unique naming convention for list name and using that name in ListMaster->createList function to create another server instance in icegrid under ListManagerApp application using server template (with activation on demand). So, these clients have 1-to-1 relationship with list and server instance.

So, I hope, I have defined my task I am working on.

Now, I have following questions:

1. Is this a good idea to create new server instance for each list, which can be started and connected through second set of clients? Idea behind this is to avoid another lookup database to find which node have that list and avoid single point of failure.

2. After adding few hundred server instance to the registry, the process slows down too much. It takes around 3 seconds to instantiateServer from template onto IceGrid registry. So, is there any other better way to do this thing, without using of another lookup registry/database? As, we are expecting to process millions of lists unlike current stuff is slowing down at couple of thousand server instance itself.

Note:
1. Its only server instantiation is slowing down, the overall performance of connection and function invocation on adapter is not slowing down.

2. Also, the startup of only 1 server instance is proportional to number of existing server instance (rest all in passive mode, i.e. not started yet). I could not understand why?

Please advice.

Many thanks.

Surya

Comments

  • benoit
    benoit Rennes, France
    Hi Surya,

    Before we look some more into this, could you upgrade to Ice 3.3.0 and see if IceGrid exhibits the same performance issues with respect to server instantiation? Note that We do not provide free support on the forums for old Ice versions.

    Also, if I understand it correctly, you have one server per table. How many nodes do you have for your application and how many servers do you deploy per node on average?

    Cheers,
    Benoit.
  • Hi Benoit,

    Since, I am working on freebsd 7, which is not your supported platform, I will have to wait for next release of port to upgrade to Ice 3.3.0

    Yes, your interpretation is right, right now I am dedicating one server instance per list (for ListManager interface), which gets dynamically created for every invocation of createList on ListMaster interface. I dedicate 1 ListMaster server per node and many ListManager server per node (which dynamically grows on number of createList invocation).

    Right now, I am in initial phase of testing the IceGrid performance for N number of ListManager Server instances (they are all set for start on-deman), where right now performance is too poor for IceGrid.Admin.instantiateServer() for N=3000. It takes around 3-4 seconds for each server instantiation.

    I am curious to find more efficient way of defining some well-known object kind of thing, to which clients can connect to same Ice Object on specific node, where list gets dynamically created. Based on this identity to this icegrid object (may be well known object or findObjectById() kind of thing), the servent will get the id of list client is requesting for and load corresponding berkeley db connection handler.

    Thanks in advance for suggesting me right artefact to use on icegrid to define this reverse proxy kind of object to keep client unaware of which node list would have been created on.

    Regards,

    Surya

    I am using Intel Quad Core, 4 Gid ram and gigabit ethernet card for server and Intel dual core with 2 gig ram and gigabit ethernet card for clients.
  • benoit
    benoit Rennes, France
    Hi Surya,

    I assume you're using the FreeBSD port from FreeBSD software?

    I'm not surprised server instantiation is getting slow with so many servers. Instantiating a server is similar to updating the application and IceGrid needs to check whether or not all the servers already deployed need to be updated. This is certainly something that could eventually be optimized in the future.

    In any case, you should perhaps consider a different approach to locate your ListManager objects. One solution would be to add one well-known object per ListManager and host multiple ListManager objects per server. You can add the well-known objects to the IceGrid registry using the IceGrid::Admin interface. Your clients will just need to create a well-known proxy with the list name to access a given the ListManager object.

    Another possibility would be to implement your own list manager registry but it's certainly a bit more work if you want it to be reliable (you can check out my article "Master–Slave Replication with Ice" from the newsletter Issue 23 for some information on how to build such a reliable registry service).

    Cheers,
    Benoit.
  • Hi Benoit,

    I am using FreeBSD standard FTP site for fetching latest ports available on FreeBSD. It might take 1-2 months for this latest Ice release to appear there. I was just wondering, how different is FreeBSD from Mac OS, as fundamentally they both have BSD origins. With little effort, you might make Ice to be portable on FreeBSD. I understand supporting is a big task, but you can provide patch without support.

    I was also wondering, whether you are using more efficient kqueue kernel event handling for MacOS which can also be used on freebsd, instead of usual "select".

    Ok, coming back to my problem. I did followed your first suggestion to create well known objects and register it to same ListManager server for each node.

    Attached is the performance stats. IceGrid Registry terminated when the ram was fully occupied.

    With the number of well-known objects increased in the registry, the icegridnode memory footprint also increased and same with the listmanager server. I was using the same listmanager object with different identity for creating new well known object, still the memory was increasing at both places (listmanager server and icegridnode).

    I did my experiment to add 30 Million well know objects into the registry and on every 2000 objects added, i did simple connection through well known object and 100 time function invocations there. Attached excel file gives nice graphs. My experiment terminated at around 7.7 Million when my RAM was full.

    I restarted icegridnode to generate attached "top" stats.

    I would like to experiment further to use better mechanism to implement my problem. I might try your second suggestion.

    Regards,

    Surya
  • benoit
    benoit Rennes, France
    Hi Surya,

    Nice graphs!

    Mac OS X and FreeBSD do indeed have few things in common. On Mac OS X, Ice uses kqueue and if not already the case, FreeBSD could also probably use it as well (by adding #ifdef for FreeBSD in the src/Ice/ThreadPool.h and ThreadPool.cpp files for 3.2.1 or src/Ice/Selector.h for 3.3.0). We have in the past incorporated changes for FreeBSD and we'll be happy to incorporate more changes if you or the FreeBSD maintainer send us these changes.

    It's not totally clear to me why the IceGrid registry memory usage increases with the number of well-known objects. Well-known objects added with the addObject method of the IceGrid::Admin interface are stored in a BerkeleyDB database so it should in theory only be limited by the number of entries a BerkeleyDB database can handle. Or are you adding the well-known objects to the application descriptor (in which case it would be expected since the application well-known objects are cached in memory)?

    To improve the memory usage of your ListManager servers, if possible you could look into using a servant locator to avoid allocating memory for each servant.

    Cheers,
    Benoit.
  • benoit wrote: »
    It's not totally clear to me why the IceGrid registry memory usage increases with the number of well-known objects. Well-known objects added with the addObject method of the IceGrid::Admin interface are stored in a BerkeleyDB database so it should in theory only be limited by the number of entries a BerkeleyDB database can handle. Or are you adding the well-known objects to the application descriptor (in which case it would be expected since the application well-known objects are cached in memory)?

    To improve the memory usage of your ListManager servers, if possible you could look into using a servant locator to avoid allocating memory for each servant.

    Thanks for moral support for FreeBSD, :) Depending on my gradually incrementing experience with Ice, I will try to bring patches in your knowledge.

    I am adding well-known object using addObject method of the IceGrid::Admin interface only. I was too expecting that if these objects are going into BDB then why memory growing for registry process? I will further investigate into it and share with you.

    I will also look into servant locator thing.

    Thanks.

    Surya