Archived

This forum has been archived. Please start a new discussion on GitHub.

glacier2 hang with a ssl error

Hi all:

Aug 02 11:14:30 54141-GLC02 glacier2router[1584]: Network: closed tcp connection
local address = xx.xx.xx.xx:49908
remote address = xx.xx.xx.xx:23913
Aug 02 11:23:14 54141-GLC02 glacier2router[1584]: connection exception:
OpenSSLTransceiverI.cpp:406: Ice::ConnectionLostException:
connection lost: Broken pipe
local address = xx.xx.xx.xx:8000
remote address = xx.xx.xx.xx:51304

glacier2 hang and icegridnode can't stop it.

Ice 3.6.2
Linux 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016

Tagged:

Comments

  • benoit
    benoit Rennes, France

    Hi,

    The connection exception warning here doesn't necessarily indicate an error. It just indicates that the connection with a client (or server) wasn't closed gracefully. It can be the result of a connection loss (network issue) or because the client crashed or forcefully closed the connection.

    The hang on shutdown of Glacier2 shouldn't occur however if you correctly set timeouts on the Glacier2 endpoints. Are you setting timeouts on the endpoints? Could you send the Glacier2 configuration you're using?

    Cheers,
    Benoit.

  • The hang is not on shutdown.
    Glacier2 do not respond to any messages include SSL handshake after this log.

    My Config:

    Configuration file (08/03/16 13:55:27.264)

    Server configuration

    Ice.Admin.ServerId=Project156.Glacier2.gnode1
    Ice.Admin.Endpoints=tcp -h localhost
    Ice.ProgramName=Project156.Glacier2.gnode1

    Server descriptor properties

    IceMX.Metrics.Debug.GroupBy=id
    IceMX.Metrics.Debug.Disabled=0
    IceMX.Metrics.ByParent.GroupBy=parent
    IceMX.Metrics.ByParent.Disabled=0
    Ice.BackgroundLocatorCacheUpdates=1
    Ice.Default.EndpointSelection=Ordered
    Ice.Default.Locator.EndpointSelection=Random
    Ice.Default.Locator.ConnectionCached=0
    Ice.Default.LocatorCacheTimeout=30
    Ice.RetryIntervals=0 10 150 200
    Glacier2.SSLSessionManager.ConnectionCached=0
    Glacier2.Client.Endpoints=ssl -h glacier01.Project156.net -p 8000
    Glacier2.Server.Endpoints=tcp -h glacier01.live.Project156.dev
    Glacier2.InstanceName=Project156.Glacier2
    Glacier2.SessionTimeout=60
    Glacier2.SSLSessionManager=RouterSessionManager
    Glacier2.Filter.Category.Accept=ClientFacade GameServerMessager
    Ice.Plugin.IceSSL=IceSSL:createIceSSL
    IceSSL.VerifyPeer=2
    IceSSL.DefaultDir=/var/lib/ice/icegrid/gnode1/servers/Project156.Glacier2.gnode1/distrib/certs
    IceSSL.CAs=cacert.pem
    IceSSL.CertFile=server.p12
    IceSSL.Password=password
    IceSSL.Trace.Security=1
    Glacier2.SSLPermissionsVerifier=Project156.Glacier2/NullSSLPermissionsVerifier
    Glacier2.Client.Trace.Reject=1
    Glacier2.Trace.Session=1
    Ice.Warn.Connections=1
    Ice.Warn.UnusedProperties=1
    Ice.Trace.Network=2
    Ice.UseSyslog=1
    Ice.MessageSizeMax=10240
    Ice.Default.Locator=Project156Grid/Locator:tcp -h sm01.live.Project156.dev -p 11000:tcp -h sm02.live.Project156.dev -p 11000:tcp -h sm03.live.Project156.dev -p 11000

  • I Found out how to reproduce the problem !!!

    After client call initiateCallback(), holding client console's scollbar to suspend client.
    After Session Destoryed 15 seconds (because Glacier2.SessionTimeout=30 ?), release the scollbar.
    Glacier is hang Now!
    if use { "_fwd", "o" }, Glacier is well.

    `public override void initiateCallback(CallbackReceiverPrx proxy, Current current__)
    {
    /Dictionary<string, string> ctx = new Dictionary<string, string> { { "_fwd", "o" } };
    proxy = CallbackReceiverPrxHelper.uncheckedCast(proxy.ice_context(ctx));
    /

        var timer = new System.Timers.Timer();
        timer.AutoReset = false;
        timer.Interval = 3000;
        timer.Elapsed += (source, e) =>
        {
            proxy.begin_callback().whenCompleted(
                () => 
                {
                    Console.WriteLine("callback completed");
                },
                (ex) =>
                {
                    Console.WriteLine($"callback exception\n{ex}");
                });
    
            SessionControlPrx control = ......;
            control.begin_destroy().whenCompleted(
                () => 
                {
                    Console.WriteLine("destroy completed");
                },
                (ex) => 
                {
                    Console.WriteLine($"destroy exception\n{ex}");
                });
        };
        timer.Start();
    }`
    
  • benoit
    benoit Rennes, France

    Hi,

    I haven't been able to reproduce so far. Could you try to attach with the debugger (gdb on Linux) to the Glacier2 router process and get a dump of the thread's stack traces?

    Cheers,
    Benoit.