Archived

This forum has been archived. Please start a new discussion on GitHub.

Tracing Exceptions in Distributed Ice Settings

Hi,

I’m looking for a way to trace exceptions over several nodes in a distributed Ice environment. This is necessary to identify the root cause of an exception observed by a client. Basically, it should simplify debugging in later development stages of our project.

To achieve this, I would like to add further information to an exception, such as the stacktrace of the server and the service endpoint where the exception occurred. Unfortunately, the server’s stacktrace of an exception is not transmitted by default. So, is there a way to include it in an exception and pass it to a client?

Furthermore, it would be useful to intercept exceptions just before they are sent to a client. This would allow us, for example, to log the exception (including its stacktrace) and to add or remove information, if necessary.

However, any other suggestions how to realise traceable exceptions based on Ice are welcome!

Thanks,
-- Jens

Comments

  • benoit
    benoit Rennes, France
    Hi,

    Which Ice version and language mapping do you use here?

    In theory, the server should already be tracing unexpected exceptions raised by a servant if you didn't set Ice.Warn.Dispatch to 0 (the default value is 1). Is there any reasons why this isn't sufficient to track down your issue?

    To intercept unexpected exceptions on the server side, you could use a dispatch interceptor. This interceptor would catch unexpected exceptions (exceptions other than the user exceptions declared for the Slice operation) raised by a servant call. It could then throw one of the unknown exception (either UnknownLocalException, UnknownUserException or UnknownException) with the "unknown" string field containing the information you need (stack trace, information on the connection).

    For more information on dispatch interceptor, see this section in the Ice manual. You can also check out the test/Ice/interceptor tests included with your Ice source distribution for some code samples.

    Cheers,
    Benoit.
  • Hi,

    we are using Ice 3.3.1 with C++ and C# language mappings.

    The system we are planning will contain multiple clients, servers, and controllers. Some services of the system will be replicated several times. They will be available at multiple servers and, in some cases, locally as well. It is transparent to a client which service instance it is using.

    If an exception is thrown (either expected or unexpected), developers need additional information in order to trace back the exception to its original source. For example, we need information about the service instance used and about its location. In the worst case, an exception is passed through multiple nodes before it is caught. In such cases, the exception should contain information about all nodes it passed and, ideally, the stack traces of all nodes as well.

    Furthermore, we have to include a customised logging mechanism (for example based on the Ice::Logger). The logger should be able to record all exceptions that are thrown by a servant (expected and unexpected ones).

    Thanks for the reference, I looked at the DispatchInterceptor as well as the UserExceptionWriter (Intercepting User Exception Insertion). Do you have some examples for the usage of the UserExceptionWriter? Would that be a reasonable point to intercept expected user exceptions?

    Cheers,
    Jens
  • benoit
    benoit Rennes, France
    Hi,

    It is not clear to me whether or not your client needs this information just for logging and debugging purposes or if they need it for their error handling logic. Not handling exceptions in middle-tier services but instead let them flow from the back-end service up to the clients also seem a bit dangerous, what can the clients do about these exceptions and won't they have to deal with potentially many kind of unexpected exceptions?

    In any case, there are no easy way to intercept user exceptions. The cleanest could be to make this explicitly part of the contract between client and services. You could for example define a base user exception for all your user exceptions, something like the following for example:
    // Slice
    module MyProject
    {
    
    struct UserExceptionFrame
    {
         string serviceId;
         string stackTrace;
    };
    sequence<UserExceptionFrame> UserExceptionFrameSeq;
    
    exception UserException
    {
         UserExceptionFrameSeq frames;
    };
    
    exception BackendException : UserException
    {
         string proxy; // Stringified proxy of the back-end Ice object.
    };
    
    }
    

    In your code, you could define some helpers to set these fields automatically and instead of letting exceptions directly propagate through the servant methods, you would add some handlers:
    // C++
    MiddleTierServantImpl::process(const Ice::Current&)
    {
        try
        {
              backendPrx->process();
        }
        catch(const MyProject::UserException& ex)
        {
              addFrameAndRethrow(ex);
        }
        catch(const Ice::LocalException& ex)
        {
              rethrowAsBackendFailure(ex, backendPrx); 
        }
    }
    

    Cheers,
    Benoit.
  • Thanks a lot Benoit, that answers my question.

    The client needs the information just for logging and debugging purposes. And, of course, free floating exceptions are dangerous. No doubt about that. However, that doesn’t mean they do not occur ;-)

    Cheers,
    -- Jens