Server side CloseConnectionException when client closes connection.

MatthewZennioMatthewZennio Member Matthew DaviesOrganization: Zennio Avance y TecnologiaProject: home automation touch panel <> mobile clients
Hi,

I have an ICE server executing on an ARM based embedded device using the full version of ICE 3.5. The aim is to connect a local client, currently a PC based emulation, to the embedded device. The connection between the two is established correctly and bi-directional communication works just fine. The problem that I am experiencing is that frequently on closing the client emulation application an Ice::CloseConnectionException is thrown on the server side which ultimately causes a SEG fault and the termination of the server application. The resulting stack trace is this:

#0  0x40bc1670 in backtrace () from /lib/libc.so.6                              
#1  0x406e1374 in (anonymous namespace)::getStackTrace() ()                     
   from /usr/lib/libIceUtil.so.35                                               
#2  0x406e2a0c in IceUtil::Exception::Exception(char const*, int) ()            
   from /usr/lib/libIceUtil.so.35                                               
#3  0x40463924 in Ice::LocalException::LocalException(char const*, int) ()      
   from /usr/lib/libIce.so.35                                                   
#4  0x404999c4 in Ice::ProtocolException::ProtocolException(char const*, int)   
    () from /usr/lib/libIce.so.35                                               
#5  0x40499ec4 in Ice::CloseConnectionException::CloseConnectionException(char c
onst*, int) () from /usr/lib/libIce.so.35                                       
#6  0x40444ebc in Ice::ConnectionI::parseMessage(IceInternal::BasicStream&, int&
, int&, unsigned char&, IceInternal::Handle<IceInternal::ServantManager>&, IceIn
ternal::Handle<Ice::ObjectAdapter>&, IceInternal::Handle<IceInternal::OutgoingAs
ync>&) () from /usr/lib/libIce.so.35                                            
#7  0x4044d31c in Ice::ConnectionI::message(IceInternal::ThreadPoolCurrent&) () 
   from /usr/lib/libIce.so.35                                                   
#8  0x406123d8 in IceInternal::ThreadPool::run(IceUtil::Handle<IceInternal::Thre
adPool::EventHandlerThread> const&) () from /usr/lib/libIce.so.35               
#9  0x4061318c in IceInternal::ThreadPool::EventHandlerThread::run() ()         
   from /usr/lib/libIce.so.35                                                   
#10 0x406f88f0 in startHook () from /usr/lib/libIceUtil.so.35                   
#11 0x40029b3c in start_thread () from /lib/libpthread.so.0


ACM is disabled on both the client and server and also retries have been disabled. All the endpoints have a timeout configured. An SSL connection is being used, but I have tried a simple TCP connection and the problem still occurs. The problem does not occur every time a connection is closed, it occurs about 1 in 3 times. I have enabled network tracing and there are no obvious issues. I have enabled the protocol trace however this seems to change the dynamics of the system so that the problem disappears.

What I have noticed is that there does not seem to be a problem if the server closes the connection. The client must 'ping' the server to maintain the connection open, so if I disable the client's network connection the server will close the connection itself which appears to work ok. The problem just seems to occur when the client closes the connection from its end (either when the ICE communicator is destroyed or if the client explicitly calls close() on the connection object).


I am not sure of the best way to fix this issue. The exception happens on an ICE thread and I don't see any means of handling the exception gracefully via the application. As a work around I am going to try adding a 'close/disconnect' method to the server's interface so that the client can inform the server to close the connection directly (without waiting to detect client inactivity). I will have to make sure that the client then gives the server enough time to close the connection before the client application exits and attempts to close the connection itself on destroying the communicator.


Before running the server on the embedded platform I had run an emulation of the server and multiple clients on a PC and there wasn't any problems closing connections from the client end.


I would be grateful if anybody has any insight into why this problem may be occurring.


Cheers

Matthew

Comments

  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi Matthew,

    What OS is your server running on?

    Can you run the server with GDB and post a stack trace of all threads when the server crash?

    Does the server crash if your disable stack traces, you can do that by setting Ice.PrintStackTraces property to 0 in your server configuration.
  • MatthewZennioMatthewZennio Member Matthew DaviesOrganization: Zennio Avance y TecnologiaProject: home automation touch panel <> mobile clients
    Hi Jose,

    Thanks for replying. I have disabled stack traces as you suggested and the server no longer crashes. I will need to do a lot more testing, but at least for the moment it looks like this issue has been resolved.

    For your information the OS is Linux 2.6.31 running on a Freescale iMX 25.

    Many thanks!
  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Seems there is some problem with the call to "backtrace" in your platform, what tool-chain are you using to build Ice?
  • MatthewZennioMatthewZennio Member Matthew DaviesOrganization: Zennio Avance y TecnologiaProject: home automation touch panel <> mobile clients
    We build ICE via LTIB (Linux Target Image Builder) using a Freescale board support package toolchain gcc-4.1.2-glibc-2.5-nptl-3.
  • xdmxdm La Coruña, SpainAdministrators, ZeroC Staff Jose Gutierrez de la ConchaOrganization: ZeroC, Inc.Project: Ice Developer ZeroC Staff
    Hi Matthew,

    I believe there is a problem with backtrace in your system, but I couldn't find any relevant bugs, maybe is related to glibc build configuration.

    You can try to update to a more recent tool-chain gcc-4.1.2-glibc-2.5-nptl-3 seems quite old.
  • MatthewZennioMatthewZennio Member Matthew DaviesOrganization: Zennio Avance y TecnologiaProject: home automation touch panel <> mobile clients
    Hi Jose,

    Yes the toolchain is pretty old, but since the OS and rootfs have proved to be stable there has not been any pressing need to update to a newer toolchain.

    My initial concern was that the server throwing the CloseConnectionException was more significant. As a result my investigation was focussed on what provoked this exception and double checking that I had not done something silly while integrating new functionality into the embedded application (also double checking my client emulation). As it appears that this exception is handled silently with no adverse behaviour when stack traces are disabled I am happy enough just to leave them disabled. I could investigate exactly why backtrace fails on our platform but at the moment developing the application has higher priority :).

    Thanks for your support, it has been very helpful.
Sign In or Register to comment.