Observers do not work properly
ignacio.diez
Member Ignacio DiezOrganization: University of Castilla-La ManchaProject: HYDRA - System Deployment ✭
in Help Center
Hello everybody,
I'm having trouble with using node observers in my project. I have a server (we'll call it HydraManager) which needs to be aware of nodes coming up and down. I set the observer via the setObserver method, and I have a thread pingin the session in order not to let it expire.
Everything works fine if the node comes up shortly after the HydraManager was started, but after some time I get no more notifications on node changes. I am using python and Ice 3.3.1-12 (also tried 3.3.1-10) in Debian. Here are some pieces of code which might be relevant:
This is the run method from HydraManger
HydraNodeObserver inherits from IceGrid.NodeObserver and implements its methods (nodeUp, nodeDown, etc). And this is the thread used to ping the session:
I can't figure out what is going on, so any help would be appreciated.
I'm having trouble with using node observers in my project. I have a server (we'll call it HydraManager) which needs to be aware of nodes coming up and down. I set the observer via the setObserver method, and I have a thread pingin the session in order not to let it expire.
Everything works fine if the node comes up shortly after the HydraManager was started, but after some time I get no more notifications on node changes. I am using python and Ice 3.3.1-12 (also tried 3.3.1-10) in Debian. Here are some pieces of code which might be relevant:
This is the run method from HydraManger
def run(self, args):
sys.stdout.flush()
c = self.communicator()
locprx = IceGrid.LocatorPrx.checkedCast(c.getDefaultLocator())
registry = locprx.getLocalRegistry()
self.session = registry.createAdminSession("manager", "passwd")
self.admin = self.session.getAdmin()
[COLOR="DarkRed"]# thread to keep the session alive[/COLOR]
self.alive = SessionThread(self.session, registry.getSessionTimeout(), "ManagerServer")
self.alive.start()
[COLOR="DarkRed"]# add the node observer[/COLOR]
self.observersAdapter = c.createObjectAdapter('ObserversAdapter')
self.observersAdapter.activate()
nodeobprx = self.observersAdapter.add(HydraNodeObserver(c), Ice.Identity("NodeObserver"))
nodeobserver = IceGrid.NodeObserverPrx.checkedCast(nodeobprx)
try:
self.session.setObservers(
None,
nodeobserver,
None,
None,
None)
except IceGrid.ObserverAlreadyRegisteredException, err:
logger.warn(err)
[COLOR="DarkRed"]# add the manager[/COLOR]
self.mgri = ManagerI()
mgrAdap = c.createObjectAdapter('ManagerAdapter')
mgrprx = mgrAdap.add(self.mgri, Ice.Identity('Manager'))
mgrAdap.activate()
logger.info("Manager ready!")
self.callbackOnInterrupt()
c.waitForShutdown()
self.interruptCallback(2)
logger.info("Exiting...")
return 0
HydraNodeObserver inherits from IceGrid.NodeObserver and implements its methods (nodeUp, nodeDown, etc). And this is the thread used to ping the session:
class SessionThread(Thread):
def __init__(self, session, timeout, name):
super(SessionThread, self).__init__()
self.session = session
self.finished = Event()
self.timeout = timeout-5
self.name = name
def run(self):
while True:
self.finished.wait(self.timeout)
if self.finished.isSet(): break
try:
self.session.keepAlive()
except Exception, err:
logger.error("%s\n%s" % (self.name, err))
def stop(self):
self.finished.set()
I can't figure out what is going on, so any help would be appreciated.
0
Comments
You should enable session tracing on the registry to see if the session if being closed by the registry. Set the IceGrid.Registry.Trace.Session property to 2 in your registry configuration file to enable this tracing. If this doesn't help, you can then try to enable networking tracing on the registry and on your python client with Ice.Trace.Network=2 to see if there are any issues with establishing the network connection from the registry to your client.
Cheers,
Benoit.
I tried what you said. First I only traced the registry, and this is what I got.
Everything looked normal, as I couldn't see the Manger.run session being destroyed nor anything strange. So I tried tracing the network; the result is in the attatched file (it was too long for a post).
I don't see anything unusual, but I'm not sure with this tracing. Perhaps you can spot some error.
Thanks again,
Nacho.
Did you try enabling protocol tracing with Ice.Trace.Protocol=2 on your client? Can you try attaching to your client process after you notice that it doesn't receive an event to verify that the Ice server thread pool threads aren't hung on some locks (if all the threads are busy, your client won't be able to dispatch further events)?
Cheers,
Benoit.
I don't know what you mean by 'attatch to your client process'. I am using python, so I cannot attatch to a process the way you do it with a C debugger.
When trying to set Ice.Trace.Protocol and Ice.Trace.Network my program won't start (without rendering any kind of message, which is most frustrating). It enters __main__, but when it hangs on the m.main(sys.argv) invocation (see below):
if __name__=="__main__": m = ManagerServer() vlevel = logging.INFO logger.setLevel(vlevel) sys.exit(m.main(sys.argv))ManagerServer is the class containing the run() method I posted on my first post.
Regards,
Nacho
You should see something similar. If this works, you can try to find out what is different in your application.
Cheers,
Benoit.
I think I have found the sources of my problem. One of them was my having several administrative sessions, one for every class that performed administrative tasks. That is solved.
The other one is some error being raised somewhere, which freezes my manager, therfore making it unable to receive events. I'm still working on this.
Thanks for your replies.