Ice.Application run method never called

lafayej

The environment is Redhat 5 + Ice3.4.1. I am running a setup of roughly 20 icegrid nodes relying in 4 icegrid registries (1 master + 3 replicas). I have been running this setup for almost one year and appreciating the reliablity of Ice when configured properly.

Last week, I had a crash of the master registry (memory exhaustion on the server, not the fault of Ice). Since then, I noticed some inconsistencies, most of which I was able to diagnose and solve myself.

Nevertheless, I have come to a tricky one which I am unable to fix. The problem is that when I run the python example below, the print statement is never reached:

---- code ----
import Ice
import sys

class TestApplication(Ice.Application):

def run(self, args):
print "**run"
self.communicator().waitForShutdown()
print "DONE"
return 0

app = TestApplication()
sys.exit(app.main(sys.argv))

---

and the ICE_CONFIG file:

---- code ----
Ice.Default.Locator=RTCSGrid/Locator:tcp -p 4061 -h rtcs-registry-master.data.cfm.fr
# Server configuration
Ice.Admin.ServerId=RTCWatch-us-1
Ice.Admin.Endpoints=tcp -h 127.0.0.1
Ice.ProgramName=RTCWatch-us-1
Ice.Trace.Network=2
Ice.Trace.Locator=3

---

Furthermore, I have noticed the following facts:
- commenting one of the Ice.Admin properties seems to fix the problem: server startup is performed and waitForShutdown is invoked. In this case, server startup from the IceGrid gui is successful but it never goes into the 'green arrow state' but stays in the 'green cross' one.
- invoking the program in a debugger gives the backtrace attached. In frame 4 (IceInternal::ConnectRequestHandler), connection variable shows a strange value: <IceUtil::HandleBase<Ice::ConnectionI>> = {_ptr = 0x32}, <No data fields>}. See attached stackstrace: Attachment not found.
- no connection to the registry is attempted, confirmed by tcpdump.

Do you have any idea which test I can run to help me troubleshoot the problem ?