How to recover from ConnectionLostException in callback?

StephanSStephanS Member Stephan SigristOrganization: Diligent IT Solutions AGProject: Network-based video surveillance
Hi,

In our application, we use sessions, bi-directional connections, and callbacks from the server to the client. From time to time, we experience the following problem. Our server tries to call back to a client but the callback fails with:
Ice.ConnectionLostException
error = 0

Surprisingly enough, the next call from the same client to our server does still work, so the underlying connection must either be still valid or transparently be re-opened.
Questions:
  1. What could cause the ConnectionLostException if the client is still there?
  2. What is the best way to inform a client that a callback has failed and a new session should be opened (otherwise, the states are not in sync anymore)?
Kind regards
Stephan

Comments

  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    When connections fail they are indeed transparently re-established under most circumstances. For the exact details you should consult the Ice manual, or read my article "Connection Management in Ice" in http://zeroc.com/newsletter/issue24.pdf.

    What would be best in your circumstance I think would be to disable automatic connection re-establishment in the client. The client is responsible for heart beating the server on a regular basis over the same connection. If the heartbeat fails, then you know the connection dropped and the client should rebuild the connection.
  • StephanSStephanS Member Stephan SigristOrganization: Diligent IT Solutions AGProject: Network-based video surveillance
    Hi Matthew

    > disable automatic connection re-establishment in the client.

    I read the article but currently, it is not obvious for me how this can be achieved. What setting/code do I have to change in the client?

    Thanks, Stephan
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    As documented on Automatic Retries, you should set Ice.RetryIntervals=-1. I also recommend disabling active connection management by setting Ice.ACM.Client=0 (see Active Connection Management for details) otherwise depending on the frequency of heart beating the connection might be closed silently by the Ice runtime and transparently re-established which is exactly what you want to avoid.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Btw, did you consider using Glacier2 instead? With Glacier2, calls from the client will automatically be rejected if the connection between the client and Glacier2 has been lost.

    If you don't wish to use Glacier2, you could implement a similar mechanism in your session servant to ensure that calls are only dispatched if they come from the connection which was used to create the session object.

    For example:
    // Slice 
    interface Session
    {
        void refresh();
        void doIt();
    };
    
    // C++
    SessionI::SessionI(const Ice::ConnectionPtr& con) : _sessionConnection(con)
    {
    }
    
    void
    SessionI::doIt(const Ice::Current& current)
    {
        checkConnection(current);
        ...
    }
    
    void 
    SessionI::refresh(const Ice::Current& current)
    {
        checkConnection(current);
        _timestamp = IceUtil::Time::now();
    }
    
    void 
    SessionI::checkConnection(const Ice::Current& current)
    {
        if(current.con != _sessionConnection)
        {
             throw Ice::ObjectNotExistException(__FILE__, __LINE__);
        }
    }
    

    With this code, you are use that calls form the client are only dispatched if they come from the connection used to create the session.

    If your session server is dealing with multiple Ice objects which are tight to the session (and underlying Ice connection) you could consider abstracting this checking in a servant locator (which is basically what Glacier2 is doing).

    Cheers,
    Benoit.
  • StephanSStephanS Member Stephan SigristOrganization: Diligent IT Solutions AGProject: Network-based video surveillance
    matthew wrote: »
    As documented on Automatic Retries, you should set Ice.RetryIntervals=-1. I also recommend disabling active connection management by setting Ice.ACM.Client=0 (see Active Connection Management for details) ...

    Hmmm, we already set these settings as follows:
    'Ice.RetryIntervals=-1'
    'Ice.ACM.Server=0'
    'Ice.ACM.Client=0'

    So I wonder, how the call from the client to the server succeeds after the ConnectionLostException in the callback. Any ideas? Has there been any changes in this area from Ice 3.3.1 to 3.4.1 (because we still use 3.3.1)?

    In addition, we also added the connection check to our session servant as suggested by Benoit to see if this improves the situation.

    Thank you, Stephan
  • matthewmatthew NL, CanadaMember Matthew NewhookOrganization: ZeroC, Inc.Project: Internet Communications Engine ✭✭✭
    I recommend you enable some tracing to find out what is going on. Try first Ice.Trace.Network=2, and also Ice.Trace.Retry=2.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    The call succeeding in the client is actually probably expected. The settings to -1 of the retry intervals disable retries if an invocation fails using a proxy associated to a working connection (i.e.: if the invocation fails after being sent or while it's being sent, the Ice runtime automatically retries to send the invocation on another connection).

    If you invoke on a proxy whose associated connection is already closed, the Ice runtime always establish a new connection first (unless you use batch requests in which case an exception is raised to notify the client that some batch requests might have been lost).

    To reliably detect the session expiration (which is the same as the session's connection being closed), the best is to implement what I suggested in my previous post.

    Cheers,
    Benoit.
Sign In or Register to comment.