High Availability with callbacks

ngambek2003ngambek2003 Member Budyanto HimawanOrganization: WMS Inc.Project: Casino back end services ✭✭
Hi,

I'm trying to write a service that is Highly Available and deploy it in multiple nodes so that if one node goes down, the request gets routed to another node.

All that works fine. I'm hitting a snag though. The client registers a callback with the service. So without any internal synchronization between the different service instances, the callback doesn't exist on the backup node.

How is this usually handled? The client is connecting to the services thru Glacier2.

Thanks
Budyanto

Comments

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Couldn't your client register the callback with the main service and the backup service?

    I don't think there's a well-known solution for this, it very much depends on the application. So if you tell us a bit more about your application we might be able to help.

    Cheers,
    Benoit.
  • ngambek2003ngambek2003 Member Budyanto HimawanOrganization: WMS Inc.Project: Casino back end services ✭✭
    Hi Benoit,

    Thanks for your response.

    The client shouldn't have to know about the backup though. Ideally the failover should be handled transparently. Which I was able to do except for the callback piece.

    The application is just a simple game. It is a java service running inside of a Java IceBox server. The game has some state which is persisted in a database. The first thing the client does is call a register method on the game, passing in a callback proxy. The game may occasionally send unsolicited messages to the client.

    I run the game on two nodes inside of IceGrid. In the middle of gameplay, I shutdown the node that is servicing the game. The proxy correctly routes the next request from the client to the second node. However, since the second node didn't get the register request, it doesn't have a reference to the callback proxy. When it comes time for the game server to send an unsolicited message back to the client, it couldn't because it has no callback proxy.

    Thanks
    Budyanto
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Yes, it's simpler if the client doesn't have to know about this backup service.

    Do you implement a Glacier2 session manager? If that's the case, it's probably best to implement this logic in your session server. The client can pass the callback to the session object created by the Glacier2 session manager. In turn, the session implementation can pass the callback object to both the main and backup services.

    If you're not using a Glacier2 session manager, I strongly recommend using it. It provides many benefits, the main ones being:
    • it shields the client from the complexity on the back-end system by providing a simple interface that the client can use to access your application functionality.
    • it provides better security as you can configure Glacier2 to make sure the client only invokes on a restricted set of Ice objects.
    • it makes it much easier to handle resource allocation on the back-end servers by the client.

    If you didn't check them out yet, I recommend to check our chat demo here and the first Ice newsletter on Glacier2 sessions here.

    Cheers,
    Benoit.
  • ngambek2003ngambek2003 Member Budyanto HimawanOrganization: WMS Inc.Project: Casino back end services ✭✭
    Thanks for the suggestion Benoit.

    I am using Glacier2 Session Manager although trivially at this point. With the session Manager approach, now the session manager is aware of the intricacies of the game.

    I haven't tried this but if Glacier2 is not in the picture would using a bidirectional connection work? What I'm thinking is that the client can pass a stringified ice_identity of the callback (instead of a proxy). The game server can store this (along with its other state) in the DB. When the game server fails over it can look up the ice_identity and create a proxy with it. In the absence of Glacier2 will this work?

    Can a similar approach be applied with Glacier2 in the picture?

    Thanks
    Budyanto
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    It's better if it's the session manager that is aware of the internals of the back-end services rather than the client. By having your client use the session object or a facade object (created by the session), you can provide a simplified view of the back-end services to your client.

    I wouldn't recommend using bi-directional connections for such a project. While it might be fine for a very simple application, going forward, Glacier2 is a much better option.

    Regarding your idea to save the identify of the client callback Ice object, this could indeed be an option but I would instead save the proxy, not just the identity. When your backup service becomes the master, it can go through this database to retrieve the proxies of the callback objects.

    A better option, which doesn't involve persisting transient proxies, would be to have your session implementation store this proxy in memory. The backup service can query all the session managers and their sessions to retrieve the callback proxies for all active sessions when needed. The advantage here is that you don't need to maintain a persistent database.

    Cheers,
    Benoit.
  • ngambek2003ngambek2003 Member Budyanto HimawanOrganization: WMS Inc.Project: Casino back end services ✭✭
    Hi Benoit,

    I didn't realize that the proxy can be persisted. As you mentioned, I thought that the proxy is "transient". I assume it has enough state that can be persisted to reconstruct itself?

    How would I persist the proxy? Is it just by serializing it?

    Budyanto
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Nothing prevents you from storing a transient proxy, as long as you understand the implications, this is fine. In particular, you need to be prepared with failures when invoking on the proxy after retrieving it from the database: the Glacier2 server might be down or have changed its server endpoints or the client session might be gone (the request will fail with a connection failure exception or with an object not exist exception).

    You can use a Freeze map to store proxies or serialize the proxy and store it in your favorite database (serialize it either as a string or as binary data using the Ice streaming APIs).

    I'm not convinced this is the best solution however as it requires to add a central/shared database which is potentially another point of failure. A design where you don't need to store this transient information would be better. As mentioned in my previous posts, you could for example have the session implementation provide the callback object to both the master and backup services.

    Cheers,
    Benoit.
  • ngambek2003ngambek2003 Member Budyanto HimawanOrganization: WMS Inc.Project: Casino back end services ✭✭
    Benoit,

    Thanks for your advice. We are exploring the different options right now. Having a centralized database is not too bad of an idea I think since our servers already need to persist some state.

    However, having to deal with the errors as you mentioned might be troublesome.

    Thanks
    Budyanto
Sign In or Register to comment.