Problem keeping IceGrid::Admin session alive

borax00

Hi,

I'm having a problem keeping an IceGrid session alive -- I've been trying to narrow it down for a while now, without too much luck. I wonder if anyone can help?

My scenario is that I have two threads:
Thread 1: is in charge of keeping the session alive,
Thread 2: starts/stops servers.

Thread 1 opens a session, then calls session->getAdmin(). It makes the IceGrid::AdminPrx available to Thread 2. Thread 1 then enters a loop in which it wakes up every sessionTimeout/2 seconds, and calls session->keepAlive(). If this fails it re-establishes the session.

Thread 2 starts and stops servers on demand.


When Thread 2 is idle, there are no problems in keeping the session alive. However when Thread 2 is fairly active, I regularly see problems. Thread 2 calls getServerState() before starting/stopping anything. Sometimes the call to getServerState() causes:
Outgoing.cpp:388: Ice::ObjectNotExistException:
object does not exist:
identity: `IceGrid/D67E7A96-032C-4D10-8DC7-647434452F44'
facet:
operation: getServerState

When this happens, the next call to session->keepAlive() (from Thread 1) causes:
Outgoing.cpp:388: Ice::ObjectNotExistException:
object does not exist:
identity: `IceGrid/4F412E29-377B-428C-AF29-00B1B0BE2F44'
facet:
operation: keepAlive

Thread 1 then re-establishes the session, but this creates a period during which Thread 2 is unable to start/stop servers.
Any ideas what might be happening here? Is this kind of thing the right design for what I want to do?


Thanks in advance,

Alex

benoit

Hi Alex,

This should work. Do you set the IceGrid.Registry.SessionTimeout property? If yes, which timeout value do you use?

Is the network connection to the registry perhaps being closed by active connection management (ACM)? (in theory it shouldn't if there's activity on the connection...).

I would enable some tracing on the client and IceGrid to try to figure out when the session is being destroyed, this might give some hints. On the client you could enable Ice.Trace.Network=2 to monitor the outgoing connection to the registry and on the IceGrid registry you can set IceGrid.Registry.Trace.Session=2 to monitor the admin session lifecyle.

Alternatively, you can also post a small self contained test case reproducing the issue, I'll be happy to take a look at it.

Cheers,
Benoit.

borax00

OK, finally got to the bottom of this -- turns out to me my fault, I had Ice.Override.Timeout set too low.

Thread 2 was trying to start a server, using the IceGrid::AdminPrx which Thread 1 was keeping alive. Because the Timeout was too low and the server took a while to start, the startServer() call would timeout, causing the IceGrid::AdminPrx to cease to be valid. Then, next time Thread 1 tried to keep it alive, it would fail.
Since the startServer() call took a long time I was calling it from a third thread, and lazily not monitoring the results (and therefore missed the timeout exception).


Thanks for the help,

Alex