Archived
glacier2 hang with a ssl error
Hi all:
Aug 02 11:14:30 54141-GLC02 glacier2router[1584]: Network: closed tcp connection
local address = xx.xx.xx.xx:49908
remote address = xx.xx.xx.xx:23913
Aug 02 11:23:14 54141-GLC02 glacier2router[1584]: connection exception:
OpenSSLTransceiverI.cpp:406: Ice::ConnectionLostException:
connection lost: Broken pipe
local address = xx.xx.xx.xx:8000
remote address = xx.xx.xx.xx:51304
glacier2 hang and icegridnode can't stop it.
Ice 3.6.2
Linux 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016
Comments
-
Hi,
The connection exception warning here doesn't necessarily indicate an error. It just indicates that the connection with a client (or server) wasn't closed gracefully. It can be the result of a connection loss (network issue) or because the client crashed or forcefully closed the connection.
The hang on shutdown of Glacier2 shouldn't occur however if you correctly set timeouts on the Glacier2 endpoints. Are you setting timeouts on the endpoints? Could you send the Glacier2 configuration you're using?
Cheers,
Benoit.0 -
The hang is not on shutdown.
Glacier2 do not respond to any messages include SSL handshake after this log.My Config:
Configuration file (08/03/16 13:55:27.264)
Server configuration
Ice.Admin.ServerId=Project156.Glacier2.gnode1
Ice.Admin.Endpoints=tcp -h localhost
Ice.ProgramName=Project156.Glacier2.gnode1Server descriptor properties
IceMX.Metrics.Debug.GroupBy=id
IceMX.Metrics.Debug.Disabled=0
IceMX.Metrics.ByParent.GroupBy=parent
IceMX.Metrics.ByParent.Disabled=0
Ice.BackgroundLocatorCacheUpdates=1
Ice.Default.EndpointSelection=Ordered
Ice.Default.Locator.EndpointSelection=Random
Ice.Default.Locator.ConnectionCached=0
Ice.Default.LocatorCacheTimeout=30
Ice.RetryIntervals=0 10 150 200
Glacier2.SSLSessionManager.ConnectionCached=0
Glacier2.Client.Endpoints=ssl -h glacier01.Project156.net -p 8000
Glacier2.Server.Endpoints=tcp -h glacier01.live.Project156.dev
Glacier2.InstanceName=Project156.Glacier2
Glacier2.SessionTimeout=60
Glacier2.SSLSessionManager=RouterSessionManager
Glacier2.Filter.Category.Accept=ClientFacade GameServerMessager
Ice.Plugin.IceSSL=IceSSL:createIceSSL
IceSSL.VerifyPeer=2
IceSSL.DefaultDir=/var/lib/ice/icegrid/gnode1/servers/Project156.Glacier2.gnode1/distrib/certs
IceSSL.CAs=cacert.pem
IceSSL.CertFile=server.p12
IceSSL.Password=password
IceSSL.Trace.Security=1
Glacier2.SSLPermissionsVerifier=Project156.Glacier2/NullSSLPermissionsVerifier
Glacier2.Client.Trace.Reject=1
Glacier2.Trace.Session=1
Ice.Warn.Connections=1
Ice.Warn.UnusedProperties=1
Ice.Trace.Network=2
Ice.UseSyslog=1
Ice.MessageSizeMax=10240
Ice.Default.Locator=Project156Grid/Locator:tcp -h sm01.live.Project156.dev -p 11000:tcp -h sm02.live.Project156.dev -p 11000:tcp -h sm03.live.Project156.dev -p 110000 -
I Found out how to reproduce the problem !!!
After client call initiateCallback(), holding client console's scollbar to suspend client.
After Session Destoryed 15 seconds (because Glacier2.SessionTimeout=30 ?), release the scollbar.
Glacier is hang Now!
if use { "_fwd", "o" }, Glacier is well.`public override void initiateCallback(CallbackReceiverPrx proxy, Current current__)
{
/Dictionary<string, string> ctx = new Dictionary<string, string> { { "_fwd", "o" } };
proxy = CallbackReceiverPrxHelper.uncheckedCast(proxy.ice_context(ctx));/var timer = new System.Timers.Timer(); timer.AutoReset = false; timer.Interval = 3000; timer.Elapsed += (source, e) => { proxy.begin_callback().whenCompleted( () => { Console.WriteLine("callback completed"); }, (ex) => { Console.WriteLine($"callback exception\n{ex}"); }); SessionControlPrx control = ......; control.begin_destroy().whenCompleted( () => { Console.WriteLine("destroy completed"); }, (ex) => { Console.WriteLine($"destroy exception\n{ex}"); }); }; timer.Start(); }`
0 -
Hi,
I haven't been able to reproduce so far. Could you try to attach with the debugger (gdb on Linux) to the Glacier2 router process and get a dump of the thread's stack traces?
Cheers,
Benoit.0