Archived

This forum has been archived. Please start a new discussion on GitHub.

Some questions about integrating glacier and grid.

kenchowcn
kenchowcn
in Help Center

Hi, I've been struggling for these questions for days, I wish experts would like to give me some valuable suggestions and I'm implementing my program based on Ice 3.6.

First, here is the template for Glacier.

<server id="${instance-name}" exe="glacier2router" activation="always">
    <properties>
        <property name="Glacier2.Client.Endpoints" value="${client-endpoints}"/>
        <property name="Glacier2.Server.Endpoints" value="${server-endpoints}"/>                
        <property name="Glacier2.InstanceName" value="${instance-name}"/>
        <property name="Glacier2.SessionTimeout" value="${session-timeout}"/>
        <property name="Glacier2.PermissionsVerifier" value="${instance-name}/NullPermissionsVerifier"/>
            <!--
        <property name="Glacier2.SessionManager" value="ChatSessionManager@ReplicatedChatAdapter"/>
            -->
            <property name="Glacier2.SessionManager" value="ChatSessionManager"/>
            <property name="Ice.Default.Locator" value="DemoIceGrid/Locator:default -h localhost -p 4061"/>
    </properties>
</server>

And this is the node.

<node name="node1">
    <server-instance template="SimpleServer" index="1"/>
    <server-instance template="Glacier2" client-endpoints="tcp -p 4063" server-endpoints="tcp"/>
</node>

I deployed this xml by icegridgui and started 200 clients, then, I checked the linux runnning status by top, I found SimpleServer-1 was doing well, but glacier2router was reaching the 100% of CPU ratio. It means that glacier2router is similar to Buckets Effect, I guess I'm not supposed to integrate glacier to the grid to limit its advantage in this situation.

However, I need glacier to do session management and callbck. I tried to seperate Glacier2 on slave, but clients have to change its Endpoints, there are tons of clients and this is definitely not a feasible idea.

Then, I think smart DNS can solve my problem, but I have to design a strong database to synchronize each glacier's node.

Finally, I wish the workflow looks like this: client-> Grid -> Glacier -> server, but the current configuration is: client-> Glacier -> Grid -> server. I understand that Glacier is for traverse and has to be in the front of any services.

Am I misundertanding anything, and is there a great solution to deal with the glacier2router in the grid structure?

I really appreciate any ideas and thanks in advance.

Comments

  • benoit
    benoit Rennes, France

    Hi,

    Glacier2 is typically the entry point to the backend system where IceGrid and application servers are isolated on a private network. You can indeed use a DNS name bound to multiple IP addresses to not have to publish all the IP adresses of the Glacier2 routers to your clients.

    What do you refer to by building a strong database? Deploying multiple Glacier2 routers and making them available through a DNS name shouldn't really require any additional development, it just requires to update the IceGrid deployment to deploy multiple Glacier2 instances.

    Finally, I'm surprise Glacier2 consumes so much CPU for 200 clients. On which platform do you run Glacier2? From which distribution did you install Ice 3.6? If you compiled Ice from sources, you should make sure to build it with optimization enabled. Is this behavior something you can easily reproduce with one of our Glacier2 demos?

    Cheers,
    Benoit.

  • kenchowcn
    kenchowcn
    edited January 2018

    Hi, Benoit.

    Building a strong database is related to our application scenarios, I think we need a strong database, but I am still considering how I can improve it somehow. Actually, we are doing a reserch for implementing heartbeat server by glacier and grid. If there are several heartbeat servers working together, then a database like redis may necessary for status inquiry.

    I forgot to descrip the 200 clients, they are simpleChat from cpp/Glacier2. Which means there are 200 users in one chatroom and each of them say i++ to all the members per second, and I deployed them by 4 virtual machine of ubuntu16.04 because that can improve terminal's productivity. let's say the server was finishing 199*199=39,601 callbacks per second without any failures except the CPU(i7-4790 CPU @ 3.60GHz) ratio reaching 100%(I tried to add one more group which is 50 clients, then forcedcloseconnections were appeared on server). The server was running Ice3.6 on ubuntu16.04 64bit and I installed them from official source by apt.

    BTW, I found however there are cpu cores I give to vitual machine for the server, Glacier2route always consumes one core. At least I tried one core and four cores, they were acting at the same performance. Did I have wrong usage?

    Thanks for your information. That help a lot.

  • benoit
    benoit Rennes, France

    Hi,

    When dealing with high message rates, you should consider disabling buffering if you don't need the features associated with it (see https://doc.zeroc.com/display/Ice37/Glacier2+Request+Buffering). This should improve a bit Glacier2 performances.

    Glacier2 or your server should be able to deal with a higher number of clients. Are you sure you aren't hitting a resource limit on your system such as the maximum number of file descriptors? You could try reduce the rate of messages send by clients and see if you can add additional clients.

    Glacier2 uses between 2 and 4 threads by default depending on whether or not buffering is enabled:

    • one thread to receive and process messages from clients (messages are queued if buffering is enabled, forwarded otherwise)
    • one thread to receive and process messages from servers (messages are queued if buffering is enabled, forwarded otherwise)
    • if client side buffering is enabled, an additional thread is used to forward client messages to the server
    • if server side buffering is enabled, an additional thread is used to forward server messages to the client

    You might be able to improve CPU usage by disabling server side buffering (Glacier2.Server.Buffered=0) and increasing the number of threads on the Glacier2 server side with Glacier2.Server.ThreadPool.Size=4 for example (the Glacier2 server side has to process and forward 200*200 messages every second). However, this will come at a price: the potential for messages to be forwarded out-of-order, see https://doc.zeroc.com/display/Ice37/Thread+Pool+Design+Considerations.

    Note that the simpleChat application wasn't optimized to allow high message rates. Designing an application which can handle high message rates is in general quite complex and requires some careful considerations on how messages are forwarded and the threading. If asynchronous messages are used without flow control for example, this can easily lead to high memory consumption and a crash when no more virtual memory is available if a server can't process quickly enough the messages sent asynchronously by the client...

    Cheers,
    Benoit.

  • kenchowcn
    kenchowcn

    Hi, Benoit,

    Thank you so much for your comprehensive reply. That really help me better understand ice.

Categories