sometime ice 3.2 method invoke run in endless loop ,cause high cpu utilization

NinaNina Member Nina ChanOrganization: PersonalProject: monitor potential exception or problem for busniess system and send alert

we are maintaining a legacy system that using ice 3.2(java) as rpc infrastructure , we using bidirectional connection in application, recently we found sometimes when client call ice_ping, the call doesn't return ,we trace java call stack ,it appear run in endless loop, causing cpu utilization greater than 80%

java.lang.Object.clone(Native Method)
java.util.Arrays.sort(Arrays.java:1206)
java.util.Collections.sort(Collections.java:157)
IceInternal.RoutableReference.createConnection(RoutableReference.java:413)
IceInternal.DirectReference.getConnection(DirectReference.java:199)
Ice._ObjectDelM.setup(_ObjectDelM.java:258)
Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1050)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:125)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)
com.dec,dd.AgentManager.testAutoServerConnection(AgentManager.java:177)

4/10 snapshots sharing following 15 elements
java.lang.Throwable.fillInStackTrace(Native Method)
java.lang.Throwable.(Throwable.java:196)
java.lang.Exception.(Exception.java:41)
java.lang.RuntimeException.(RuntimeException.java:43)
java.lang.ClassCastException.(ClassCastException.java:39)
Ice.ObjectAdapterI.isLocal(ObjectAdapterI.java:576)
IceInternal.ObjectAdapterFactory.findObjectAdapter(ObjectAdapterFactory.java:193)
Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1038)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:125)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)
com.dec,dd.AgentManager.testAutoServerConnection(AgentManager.java:177)

we think there is a ConnectionI in wrong state ,it 's _ exception field is not null , but state is not close,

is there anyone encounter the problem or is it known bug , please help, thank you !

Tagged:

Answers

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    Hi,

    Note that we don't provide free support for such old Ice versions, the best would be to upgrade to the latest Ice version. In any case, I don't see why Ice would indefinitely loop here. Can you verify that it's not your code which is actually looping calling ice_ping? You could for example add some tracing in the implementation of dd.AgentManager.testAutoServerConnection.

    Cheers,
    Benoit.

  • NinaNina Member Nina ChanOrganization: PersonalProject: monitor potential exception or problem for busniess system and send alert

    hi, benoit
    we have checked our code , we don't loop calling ice_ping in our code ,we found there is indefinite loop in ice_ping method ,as follows :
    private void ice_ping(Map __context, boolean __explicitCtx)
    {
    if ((__explicitCtx) && (__context == null))
    {
    __context = _emptyContext;
    }

    int __cnt = 0;
    for (;;)
    {
    _ObjectDel __del = null;
    try
    {
    __del = __getDelegate();
    __del.ice_ping(__context);
    return;
    }
    catch (LocalExceptionWrapper __ex)
    {
    __cnt = __handleExceptionWrapperRelaxed(__del, __ex, __cnt);
    }
    catch (LocalException __ex)
    {
    __cnt = __handleException(__del, __ex, __cnt);
    }
    }
    }

    when calling ice_ping, two exceptions throw ,one is ClassCastException thrown by findObjectAdapter method , its message is IceInternal.DirectReference cannot be cast to IceInternal.IndirectReference ,another exception is LocalExceptionWrapper, it thrown by sendRequest method, and it wraps a TimeoutException, the ice_ping method catch the ExceptionWrapper, and retry ,but the sendRequest still throws LocalExceptionWrapper, so ice_ping indefinitely loop。but we can't find out cause of the problem,more exception is:
    java.lang.Throwable.fillInStackTrace(Native Method)
    java.lang.Throwable.(Throwable.java:181)
    java.lang.Exception.(Exception.java:29)
    IceInternal.LocalExceptionWrapper.(LocalExceptionWrapper.java:16)
    Ice.ConnectionI.sendRequest(ConnectionI.java:534)
    IceInternal.Outgoing.invoke(Outgoing.java:72)
    Ice._ObjectDelM.ice_ping(_ObjectDelM.java:67)
    Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:126)
    Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)

    could you give me some troubleshooting advice , we can't upgrade ice at present

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    Can you try to enable Ice.Trace.Retry=1 and Ice.Trace.Network=2 to see if some tracing is printed out when this loop occurs?

    Cheers,
    Benoit.

Sign In or Register to comment.