Archived

This forum has been archived. Please start a new discussion on GitHub.

Dynamic Monitoring

ZeroC,

First of all I would like to congratulate to create such a wonderful product.

If I have a ICE server which for whatever reason has to restart every day, and I want the clients to reconnect when its started up again what is the best way to do it? I was thinking of broadcasting an IceStorm message "started" to the clients first. I believe IceGrids dynamic monitoring was meant to be used for this, am I right? I am guessing IceGrid would use IceStorm internally for this.

I tried to test this in python but I am not getting the applicationAdded messages, do I need to register and de-register for the messages to be sent out or they are automatic (based on the timeout)? For simplicity I do not need the IceGrids deployment feature.

I am including the simple test app in the attachment.

Thank you,
Joe

Comments

  • matthew
    matthew NL, Canada
    Overall the simplest method is not to force the client to do anything. That is the server maintains persistent state, which is re-established on startup. Exactly how you can make this work really depends on the interaction patterns of your client/server. Are the clients passive or active entities? That is, does the client interact with the server, or does the server interact with the client?
  • Matthew,

    Thank you for your reply. Unfortunately the server is just a "messenger" and the client needs to send multiple initialization calls (active) to the "real server" which is out of my control. It is not even possible to store the state in the messenger as the state of the real server changes on restart. ICE would fit in nicely where the messenger would just serve as a layer to hide the communication logic for the clients.

    I see two solutions:
    1) keep the messenger running, implement the broadcast myself, implement the detection of real server coming live
    2) startup the messenger with the real server, inform the clients that they can reconnect using IceStorm or IceGrid

    Thank you,
    Joe
  • matthew
    matthew NL, Canada
    If the clients are active in that the client calls on the server, then won't they discover that they are no longer connected? If the clients are purely passive, can't you discover that they are no longer connected because they don't get any activity?
  • Matthew,

    OK this is how it works: The client has to register and request some data. The server will start streaming back the data which the client will monitor. Occasionally the client needs to do some action based on the data. So the client is a re-active type.

    I am using one-way connections because the Ice deadlocks in the callbacks (I started playing with the thread count but oneway-ing was the easiest solution), so in this case I wont get any notification about the server being dead.

    It does not make any sense to me to implement my logic if ICE has such a feature already implemented, I have found IceStorm and IceGrid which I could use. I spent the whole weekend trying to make the dynamic monitoring work in IceGrid but I couldn't :(

    My question was whether I can use the dynamic monitoring as it seemed to be a perfect fit for what I needed it for. The documentation is limited for this feature. Do you see any problem with the code example I have sent you? Did I misunderstand what the dynamic monitoring is or what it can do ?

    Thank you,
    Joe
  • matthew
    matthew NL, Canada
    am using one-way connections because the Ice deadlocks in the callbacks (I started playing with the thread count but oneway-ing was the easiest solution), so in this case I wont get any notification about the server being dead.

    You really want to use oneway messages when you have oneway message semantics, not to avoid a deadlock. You can avoid the deadlock by suitable configuration (see demo/Ice/callback for an example), or by using AMI on the server side.

    Your server also looks quite a bit like IceStorm if you consider the interaction model of a subscriber with IceStorm. If you look closely at IceStorm you'll see that it keeps the subscriber proxies in a persistent database. The primary reason for this is that without that an IceStorm crash would mean a dead system. Subscribers are generally passive, and therefore cannot easily distinguish between an inactive IceStorm, and a dead IceStorm.
    My question was whether I can use the dynamic monitoring as it seemed to be a perfect fit for what I needed it for. The documentation is limited for this feature. Do you see any problem with the code example I have sent you? Did I misunderstand what the dynamic monitoring is or what it can do ?

    What dynamic monitoring feature are you talking about in IceGrid precisely? IceGrid has facilities for automatic activation of servers, is that what you mean?
  • Matthew,
    You really want to use oneway messages...

    You are right, reverting back to twoway is on my TODO list, it was just a dirty hack (felt “safe” as the server and client are on the same box). Thank you for confirming that AMI is what I need. With twoway, I will see the disconnect but should I try reconnecting in a loop until the server is back up again (about 15 minutes)?
    Your server also looks quite a bit like IceStorm...

    IceStorm has ICE publishers and subscribers. The server I need to communicate with via sockets is a third party (without source), I cannot put ICE code in it. To hide the communication complexity with it I have created a proxy server (named messenger previously). The proxy sever is like half IceStorm, yes. The difference is that the state I would need to persist cannot be persisted as it changes when the real server is restarted. The clients need to re-negotiate and re-establish the new state. I planned to use IceStorm for reconnect notification unless Dynamic Monitoring would work.
    What dynamic monitoring feature are you talking about...

    No, I need manual startup (DynamicRegistration).
    There is a small chapter on it in the documentation under IceGrid/Administrative Sessions/Dynamic Monitoring (35.14.3 in v3.3.0). Does Dynamic Monitoring only work with automatic activation ?

    Thank you,
    Joe
  • matthew
    matthew NL, Canada
    You are right, reverting back to twoway is on my TODO list, it was just a dirty hack (felt “safe” as the server and client are on the same box). Thank you for confirming that AMI is what I need. With twoway, I will see the disconnect but should I try reconnecting in a loop until the server is back up again (about 15 minutes)?

    Here you talking about having the client reconnect to the server? How will your client know the server is gone? What I've been suggesting is that the server keeps in persistent state the current set of connected clients. In this way, the clients won't need to reconnect when the server comes back up. However, perhaps this isn't practical for your application.
    No, I need manual startup (DynamicRegistration).
    There is a small chapter on it in the documentation under IceGrid/Administrative Sessions/Dynamic Monitoring (35.14.3 in v3.3.0). Does Dynamic Monitoring only work with automatic activation ?

    Thank you,
    Joe

    Ok, that IceGrid dynamic monitoring does not work with manually started servers, it only works with IceGrid managed servers. So in short, you must either:

    - Keep persistent state in your server so that the clients do not need to re-register (as IceStorm does).
    - Deploy your server with IceGrid and use the IceGrid observer interfaces as described in section 35.14.3
    - Implement your own monitoring service for your server.
    - Have your client monitor the server status on its own, reconnecting when necessary. You could do this by using sessions; see demo/Ice/session for an example.
  • Thank you.

    Joe