Home Help Center

Glacier2, IceGrid, and Load Balancing

sethseth Member sethOrganization: seth piezasProject: demo
I'm trying to figure out the deployment structure of a glacier secured IceGrid. With multiple routers in front of both the registry and the server nodes, does this imply that a client would have to open a session to the registry to locate an object and an ADDITIONAL session to the server to reach the object? Or would the same session be used to retrieve server objects?

It seems that with a single session, a single router could become innundated - requiring a load balancing mechanism to distribute sessions among routers.

Thanks,

Comments

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    You need only one session. The same session can be used to access the IceGrid registry or objects running on multiple backend servers (which can be spread on many machines).

    Typically, the Glacier2 router is on a machine which has access to the backend servers and can be accessed from the "outside", it doesn't have to run on the same machines as the backend services and servers.

    Cheers,
    Benoit.
  • sethseth Member sethOrganization: seth piezasProject: demo
    thanks

    thanks, benoit.

    the reason that we were thinking of putting a router on the servernode was to reduce network traffic. if the process is indeed extremely lightweight, then it seemed like it could be bundled on each server.

    failure would also be highly atomic.

    does it sound like a good idea? also, would you be able to send a link to your formal support services?
  • marcmarc FloridaAdministrators, ZeroC Staff Marc LaukienOrganization: ZeroC, Inc.Project: The Internet Communications Engine ZeroC Staff
    seth wrote:
    also, would you be able to send a link to your formal support services?

    Please contact us at [email protected] for commercial support services.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    You can deploy multiple Glacier2 router instances to spread the load. However, typically, a client will only connect to one Glacier2 router instance and this router instance will forward the Ice requests on behalf of the client to the backend Ice servers regardless of their location on the network.

    Although it's possible, I don't think it's a good idea to deploy one Glacier2 router instance per host where your servers are located and have the client to connect to each router instance to have access to the host servers. This would be too complicated to manage in the client and error prone (you'd have to configure different router proxies depending on the servers you wan't to talk to).

    Glacier2 is similar to a firewall, it sits between the public and private network and all the traffic goes through it. Note that Glacier2 is not the only way to secure your Ice application, depending on your requirements, you might also just be able to use IceSSL.

    Cheers,
    Benoit.
  • sethseth Member sethOrganization: seth piezasProject: demo
    glacier2 clients not initiating load balancing

    thanks for the update. i'll check out icessl as well.

    my current hurdle is that i've wired a simple scenario to test load balancing and replication groups that doesn't seem to work when coupled with glacier2.

    the icegrid server seems to operate fine. as does a local client. each time the app requests a proxy (with the rep group in round robin mode), the next server is activated.

    however, clients that go through a glacier2 router never initiate the grid to retrieve the next round robin server. they do retrieve an object, but it is always from the same server. this behaviour looks router bound because restarting the router initiates a new activation (once the client connects) that remains for the lifetime of the router.

    i've done all of this concurrently with multiple clients behind and in front of the router running at the same time. in all cases, requests that travel through the router never initiate more than one activation.

    ?

    thanks again,
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Yes, I'm afraid it's a current limitation when using Glacier2 and IceGrid, replica group load balancing doesn't work well. Let me try to explain the reason.

    When using Glacier2, your client never uses the IceGrid registry locator. All the locator requests are done by Glacier2 on behalf of the client. Also, your client doesn't connect directly to the server, it's Glacier2 which connects to the server on behalf of the client (Glacier2 acts as a connection concentrator).

    So when a client invokes on an indirect proxy from a replica group, Glacier2 queries the locator to retrieve the endpoints of the replica group. If multiple endpoints are retrieved, Glacier2 connects to one of the endpoint or re-use an existing connection if a connection to one of the endpoints is already established.

    When another client makes an invocation on the same proxy, Glacier2 will simply re-use the existing connection -- it won't invoke on another replica.

    Ideally, you should be able to tell Glacier2 to not re-use an existing connection (this can be achieved by creating a proxy with the ice_connectionCached() proxy method however there's currently no way to tell Glacier2 to use such a proxy). Improving this is on our TODO list however!

    Cheers,
    Benoit.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Btw, you could also consider implementing a facade object that your clients would use to access the functionallity provided by your servers. See [post=5793]this post[/post] or [post=2212]this post[/post] for more information on facade objects in the context of Glacier2. Here, the implementation of your facade object would take care of the load balancing issues and this way your clients wouldn't have to worry about it.

    You might also want to take a look at the "Session Management" article in the first newsletter for more information on Glacier2 sessions.

    Cheers,
    Benoit.
  • sethseth Member sethOrganization: seth piezasProject: demo
    shoot...

    hi benoit,

    thanks for the info. and the pointers.

    i was hoping that i was doing something wrong. it seems that this limitation pretty much destroys the integrity of the whole grid system, at least for web services. i would hate to have to code around this limitation if it's a high priority bug. do you know when it would be tended to?

    seth
  • marcmarc FloridaAdministrators, ZeroC Staff Marc LaukienOrganization: ZeroC, Inc.Project: The Internet Communications Engine ZeroC Staff
    I believe a real-world application should use the facade approach. There are several security implications if you do not use a facade, but instead let the client directly decide how Glacier2 should handle connections. Glacier2 is foremost a firewall, that should hide the backend implementation from the clients as much as possible.

    If you still want to use the other approach, and have a commercial need for this feature, please contact us at [email protected] to discuss this.
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    This, btw, is the topic of the recent set of articles that I've been writing for Connections. The next issue will have another article on this topic. I encourage you to read them to gain some insight about the security implications of letting potentially untrusted applications gain direct access to the grid without some form of facade.
  • sethseth Member sethOrganization: seth piezasProject: demo
    sessions

    thanks. i've actually been going through the articles and the documentation. for some reason, even with a facade object i can't initiate load balancing behaviour. this is incorrect behaviour i believe. i should be seeing load balancing, correct?

    the pseudo code now is:

    from Client:
    session = router.createSession(id, pw)
    session.sayHello()

    where session is an object hosted by a session server behind glacier2. sayHello() now retrieves a hello object from one of three servers from the IceGrid simple example with

    hello = Demo.HelloPrx.checkedCast(self.communicator().stringToProxy("hello"))
    hello.sayHello()

    under these conditions, even though the client communicates with glacier2, the session server should have direct access to the registry. correct?
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    If you do it that way, then yes. However, the point generally in a facade is that you do not permit such direct access to the back-end servers because its unsafe to do so.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    The reason why you don't get any load balancing is that by default Ice caches the connection with the proxy (or re-use in priority an existing one) as well as the endpoints retrieved from the locator. To disable this caching, you should create the proxy with:
    Ice.ObjectPrx obj = self.communicator().stringToProxy("hello");
    obj = obj->ice_connectionCached(false); // Disable proxy connection caching.
    obj = obj->ice_locatorCacheTimeout(0); // Disable the use of the locator cache.
    hello = HelloPrx.uncheckedCast(obj);
    hello.sayHello()
    

    With these settings, each call on the hello proxy will cause the Ice client runtime to query the locator to get the endpoints and the Ice client runtime will also eventually open connections on all the replicas.

    Cheers,
    Benoit.
  • sethseth Member sethOrganization: seth piezasProject: demo
    caching

    thanks, benoit.

    it took a while to figure that those settings. i went through the documentation in the icegrid section over and over, but found it elsewhere. i might have missed it, but if i haven't it might be useful to include that info explicity in the load balancing section.

    session.sayHello() now load balances correctly and i'm sure i can use that info to make

    hello = session.getHello()
    hello.sayHello()

    balance the same way
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    I agree, we'll look into improving the documentation regarding these settings.

    As for:
      hello = session.getHello()
      hello.sayHello()
    

    This won't work (assuming sayHello() is a call going through Glacier2). As explained in a previous post, there's no way to specify the connection caching and locator cache timeout settings to Glacier2 (these settings are not transmitted over the wire and there's no Glacier2 forwarding options for them).

    Cheers,
    Benoit.
Sign In or Register to comment.