Archived

This forum has been archived. Please start a new discussion on GitHub.

sometime ice 3.2 method invoke run in endless loop ,cause high cpu utilization

Nina
Nina
in Help Center

we are maintaining a legacy system that using ice 3.2(java) as rpc infrastructure , we using bidirectional connection in application, recently we found sometimes when client call ice_ping, the call doesn't return ,we trace java call stack ,it appear run in endless loop, causing cpu utilization greater than 80%

java.lang.Object.clone(Native Method)
java.util.Arrays.sort(Arrays.java:1206)
java.util.Collections.sort(Collections.java:157)
IceInternal.RoutableReference.createConnection(RoutableReference.java:413)
IceInternal.DirectReference.getConnection(DirectReference.java:199)
Ice._ObjectDelM.setup(_ObjectDelM.java:258)
Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1050)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:125)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)
com.dec,dd.AgentManager.testAutoServerConnection(AgentManager.java:177)

4/10 snapshots sharing following 15 elements
java.lang.Throwable.fillInStackTrace(Native Method)
java.lang.Throwable.(Throwable.java:196)
java.lang.Exception.(Exception.java:41)
java.lang.RuntimeException.(RuntimeException.java:43)
java.lang.ClassCastException.(ClassCastException.java:39)
Ice.ObjectAdapterI.isLocal(ObjectAdapterI.java:576)
IceInternal.ObjectAdapterFactory.findObjectAdapter(ObjectAdapterFactory.java:193)
Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1038)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:125)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)
com.dec,dd.AgentManager.testAutoServerConnection(AgentManager.java:177)

we think there is a ConnectionI in wrong state ,it 's _ exception field is not null , but state is not close,

is there anyone encounter the problem or is it known bug , please help, thank you !

Tagged:

Comments

  • benoit
    benoit Rennes, France

    Hi,

    Note that we don't provide free support for such old Ice versions, the best would be to upgrade to the latest Ice version. In any case, I don't see why Ice would indefinitely loop here. Can you verify that it's not your code which is actually looping calling ice_ping? You could for example add some tracing in the implementation of dd.AgentManager.testAutoServerConnection.

    Cheers,
    Benoit.

  • Nina
    Nina

    hi, benoit
    we have checked our code , we don't loop calling ice_ping in our code ,we found there is indefinite loop in ice_ping method ,as follows :
    private void ice_ping(Map __context, boolean __explicitCtx)
    {
    if ((__explicitCtx) && (__context == null))
    {
    __context = _emptyContext;
    }

    int __cnt = 0;
    for (;;)
    {
    _ObjectDel __del = null;
    try
    {
    __del = __getDelegate();
    __del.ice_ping(__context);
    return;
    }
    catch (LocalExceptionWrapper __ex)
    {
    __cnt = __handleExceptionWrapperRelaxed(__del, __ex, __cnt);
    }
    catch (LocalException __ex)
    {
    __cnt = __handleException(__del, __ex, __cnt);
    }
    }
    }

    when calling ice_ping, two exceptions throw ,one is ClassCastException thrown by findObjectAdapter method , its message is IceInternal.DirectReference cannot be cast to IceInternal.IndirectReference ,another exception is LocalExceptionWrapper, it thrown by sendRequest method, and it wraps a TimeoutException, the ice_ping method catch the ExceptionWrapper, and retry ,but the sendRequest still throws LocalExceptionWrapper, so ice_ping indefinitely loop。but we can't find out cause of the problem,more exception is:
    java.lang.Throwable.fillInStackTrace(Native Method)
    java.lang.Throwable.(Throwable.java:181)
    java.lang.Exception.(Exception.java:29)
    IceInternal.LocalExceptionWrapper.(LocalExceptionWrapper.java:16)
    Ice.ConnectionI.sendRequest(ConnectionI.java:534)
    IceInternal.Outgoing.invoke(Outgoing.java:72)
    Ice._ObjectDelM.ice_ping(_ObjectDelM.java:67)
    Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:126)
    Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)

    could you give me some troubleshooting advice , we can't upgrade ice at present

  • benoit
    benoit Rennes, France

    Can you try to enable Ice.Trace.Retry=1 and Ice.Trace.Network=2 to see if some tracing is printed out when this loop occurs?

    Cheers,
    Benoit.

Categories