glacier2 reusing the wrong connection with multiple servers of the same type?

nefilimnefilim Member Peter vanRensburgOrganization: LockerzProject: Social Platform
hi

i'm experiencing a problem with glacier2 (ice 3.3.1) where i have a client communicating with multiple proxies behind the glacier2 router. most cases are working just fine except for when trying to communicating with the same type of proxy.

the client is a multi threaded process with a single default router and single session set at the communicator level:
            Ice.RouterPrx defaultRouter = Dashboard.communicator().getDefaultRouter();
            router = Glacier2.RouterPrxHelper.checkedCast(defaultRouter);
            log.info("created glacier2 router " + router);
            session = router.createSession("peter", "peter");
            log.info("created glacier2 session " + session);

behind the router i have multiple services (not using icegrid) of the same type on different servers (but on the same port).
the requests to these completes fine but all effectively only communicating with the first proxy, all the requests returns exactly the same values.

when enabling network tracing on the glacier2 router i can see the connection getting established to the first proxy but the next incoming calls to the other proxies of the same type does not establish connections to their discrete addresses but rather reuse the first connection established (which is the wrong IP!).

the glacier2 config is minimal:
Glacier2.Client.Endpoints=tcp -h 10.208.242.145 -p 4063
Glacier2.SessionTimeout=60
Glacier2.CryptPasswords=ice.glacier.passwords
Ice.Trace.Network=3

ACM is disabled in the client, even though i can't see this being related to the problem since it is discrete endpoints. also, if i print out the proxies on the client side, the stringified proxy does reflect the discrete endpoints.

is this a problem or limitation with glacier2?

thanks.

Comments

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    It's not clear to me what you mean by "discrete endpoints".

    By default the Ice runtime always try to re-use an already established connection when invoking on a proxy with multiple endpoints. It's possible to disable this behavior by calling ice_connectionCached(false) on the proxy or with the proxy property "ConnectionCached".

    However, this won't work in your case since you're going through Glacier2. Setting this on the client proxy won't have any effect because the client doesn't directly invoke on the server, Glacier2 invokes on the server. Unfortunately, there are currently no ways to change the default settings of proxies used by Glacier2 to invoke on the backend servers (other than using the Ice.Default.<name> properties, but there's currently no Ice.Default.ConnectionCached property).

    Your client will need to create multiple proxies from the proxy with multiple endpoints and invoke on one of them if you want the request from the client to be sent on a random backend server.

    I'll add something to our TODO list to improve this.

    Cheers,
    Benoit.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Btw, this is typically not an issue when your client uses a facade/session object instead of directly invoking on the backend servers.

    The client invokes on the facade object (which is hosted by your session server behind Glacier2) and the implementation of the facade invokes on the backend servers on behalf of the client. The implementation of this facade object can easily set custom settings on the proxies with multiple endpoints to enable per-request load balancing for example.

    The session/facade object provides many advantages, it provides a simpler view of the back-end system to the client, it shields your client from changes on the back-end servers (deployment changes, interface changes, etc) and provides greater security (you can configure Glacier2 to restrict your client to only invoke on this object).

    For an example of a Glacier2 client using a session, see the Glacier2/chat demo included in your distribution or our chat demo.

    Cheers,
    Benoit.
  • nefilimnefilim Member Peter vanRensburgOrganization: LockerzProject: Social Platform
    hi benoit

    appreciate all the information. unfortunately i dont think i explained the problem well enough. let me try to elaborate:

    i have a single multi threaded client C1, with a default glacier router G1 (which is bound to the internal network, tunnel through firewall to it).
    behind G1 i have 3 servers, A, B, C each one running UserService on port 10350. (this was what i meant by discrete, the same ice application on the same port but on three separate servers/IPs).

    when i invoke the same method on the proxy for A.UserService, G1 will establish a connection to A:10350 and return the correct results, then when invoking the same method on the proxy for B.UserService, it uses the previously established connection to A:10350 and returns the same (but now incorrect) results. the same happens for the invocation on C. the net results is that all invocations happens to A.UserService.

    i hope that explains the situation a bit better. i can add some glacier trace.network logs and client debug output to further illustrate the problem if need be.

    thanks!
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi Peter,

    Are you by any chance using the same identity for the Ice object on A, B and C (you specify the identity when your register the servant with the object adapter in the server)? Identities must be unique and it's especially important for Glacier2 which rely on the identity of the Ice objects to route requests from the client to the back-end servers.

    Cheers,
    Benoit.
  • nefilimnefilim Member Peter vanRensburgOrganization: LockerzProject: Social Platform
    hi benoit

    thank you for quick answer, you are indeed correct, i do use the same identity on the server side:
                userServiceAdminAdapter.add(userServiceAdmin, Ice.Util.stringToIdentity("UserServiceAdmin"));
    

    i was thinking that glacier2 makes the routing decision based on the supplied proxy information (identity + IP + port) rather than the identity alone?
    from what you are saying, does this mean i have to uniquely qualify each identity (by adding the ip for example) for each instance of the same service i'm running? it sounds counter intuitive to the way that the ice run time is able to work correctly with multiple endpoints (Identity: tcp -h host1 -p 10:tcp -h host2 -p 10). i haven't thought through this completely yet but i wonder if i will always be able to know the uniquely qualified identity from the client perspective.

    thanks much
    peter
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Yes, the routing is based on the identity.

    Using the IP address in the identity might not be a good idea if the machine hosting your Ice servers changes of IP address (or if you want to move your servers to another machine).

    Typically, clients use the Ice location service to locate Ice objects by identity or by type. This way they don't need to know anything about the location of your Ice servers In your case, you could use IceGrid to register your 3 UserServiceAdmin objects and your clients would use the IceGrid::Query interface to lookup all user service admin objects (by type).

    For more information on this, I recommend you to read Teach yourself IceGrid in 10 minutes.

    Cheers,
    Benoit.
  • nefilimnefilim Member Peter vanRensburgOrganization: LockerzProject: Social Platform
    thanks for your suggestion Benoit. i finally got some time to look into this and got a proof of concept setup, with load balanced replica group:
            <replica-group id="UserServerGroup">
                    <object identity="UserService" type="::com::lockerz::phoenix::slice::UserService"/>
                    <load-balancing type="adaptive" load-sample="1" n-replicas="0"/>
            </replica-group>
    

    IceGrid/Query.findAllObjectsByType("::com::lockerz::phoenix::slice::UserService") returns a single proxy, the replica group, UserService -t @ UserServerGroup.

    it looks like if i use IceGrid/Query.findAllReplicas(findObjectByType("::com::lockerz::phoenix::slice::UserService")) i get a list of proxies for all the adapters (one from each node in my case), which is what i wanted and appears to continue to work through glacier2!

    i was wondering if this is the right way to go about it and if there's a faster way (1 vs 2 calls to IceGrid/Query)?

    thanks
    peter
  • nefilimnefilim Member Peter vanRensburgOrganization: LockerzProject: Social Platform
    unfortunately it looks like i spoke too soon, still experiencing the exact same problem going through glacier2 as before. a snippet from my test code output (i expect slightly different values for total jvm heap size):
    2010-02-16 12:23:51,256 [main] INFO User.TestGridClient - created glacier2 router Glacier2/router -t:tcp -h 192.168.100.3 -p 4063
    2010-02-16 12:23:51,262 [main] INFO User.TestGridClient - created glacier2 session null
    found 2 proxies
    found proxy UserServiceAdmin -t @ UserServer1.UserServiceAdminAdapter
    total jvm heap on server: 32112640
    found proxy UserServiceAdmin -t @ UserServer2.UserServiceAdminAdapter
    total jvm heap on server: 32112640
    

    my service has two adapters, the second for admin purposes, it's these adapters i'm trying to reach individually (while the services provided are stateless i do have some state (reference cache) that needs to be reloaded on each node).
    here are some of the relevant parts of my grid configuration:
            <replica-group id="UserServerGroup">
                    <object identity="UserService" type="::com::lockerz::phoenix::slice::UserService"/>
                    <load-balancing type="adaptive" load-sample="1" n-replicas="0"/>
            </replica-group>
    
            <replica-group id="UserServerAdminGroup">
                    <object identity="UserServiceAdmin" type="::com::lockerz::phoenix::slice::UserServiceAdmin"/>
                    <load-balancing type="round-robin"/>
            </replica-group>
    
            <server-template id="UserServerTemplate">
                    <parameter name="index"/>
                    <parameter name="exepath" default="java"/>
                    <server id="UserServer${index}" exe="${exepath}" activation="manual">
                            <env>CLASSPATH=${DefaultClasspath}</env>
    <option>com.lockerz.phoenix.user.UserServiceMain</option>
                            <adapter name="UserServiceAdapter" endpoints="tcp -p 10000" replica-group="UserServerGroup"/> 
                            <adapter name="UserServiceAdminAdapter" endpoints="tcp -p 10050" replica-group="UserServerAdminGroup"/>
                    </server>
            </server-template>
    

    looking at the network trace in glacier2 i can see that it reuses the first established connection for UserServer1.UserServiceAdminAdapter when communicating with UserServer2.UserServiceAdminAdapter :(

    can you see any problems with my configuration/usage or is this an Ice limitation?

    also, just as a sanity check, if i remove the router from the client configuration and specify a Ice.Default.Locator directly in the client, it establishes discrete and correct connections for each of the two adapters and behaves as expected.

    thanks
    peter
Sign In or Register to comment.