Glacier2 cannot reach PermissionVerfier on server

janos

We have a simple setup of an IceGrid including a templated server and a Glacier2 instance inside. Everything works smoothly for some days, but after having the server run a long time, the Glacier2 instance cannot access the PermissionVerifier hosted on the server. Technically there are no changes happening in the background, there is only one network interface on the machine and DHCP is disabled by default. Is there any reason why the Glacier instance cannot reach the server? The server is running and functional - as the built-in REST API can still be reached and responds within normal call times.
Is it a configuration issue on the Glacier2 instance? Do I need to have a permanent connection to the backend server set via the ACM (Ice.Glacier2.ACM.Server.Close=0)?

Thanks in advance

xdm

Hi Janos,

You don't need a persistent connection with the permission verifier. The best way to diagnostic this is to enable network and protocol tracing in Glacier2 router. You can add the following lines to your Glacier2 router configuration to enable tracing:

Ice.Trace.Network=3
Ice.Trace.Protocol=1

Can you also specify what Ice version are you using?

janos

Hi xdm,

Here's the error log (suprisingly printed via the Ice.StdOut property) from the Glacier2 instance (Ice.Warn.Connections=3):

-! 04/07/17 18:54:30.976 Test.Glacier2: warning: unable to contact permissions verifier `[email protected]'
   Reference.cpp:1637: Ice::NoEndpointException:
   no suitable endpoint available for proxy `Test.CPermissionVerifier -t -e 1.1 @ Test-Backends'
-! 04/07/17 18:54:30.981 Test.Glacier2: warning: unable to contact session manager `[email protected]Test-Backends'
   Reference.cpp:1637: Ice::NoEndpointException:
   no suitable endpoint available for proxy `Test.CSessionManager -t -e 1.1 @ Test-Backends'

Unfortunately the Ice.StdErr and Ice.StdOut properties were not enabled at the time and I cannot reproduce the problem myself. I enabled logging on the router, when it happens again during the week, I will post further details.
I thought the server checked out and removed it's endpoints, but I had no information in the server logs.

Does this help you any further?

xdm

For some reason IceGrid returns empty endpoints for the permission verifier/session manager.

Can you enable additional trace for IceGrid:

IceGrid.Registry.Trace.Locator=1

And Glacier2:

Ice.Trace.Locator=1