sometime ice 3.2 method invoke run in endless loop ,cause high cpu utilization

NinaNina Nina ChanOrganization: PersonalProject: monitor potential exception or problem for busniess system and send alertMember

we are maintaining a legacy system that using ice 3.2(java) as rpc infrastructure , we using bidirectional connection in application, recently we found sometimes when client call ice_ping, the call doesn't return ,we trace java call stack ,it appear run in endless loop, causing cpu utilization greater than 80%

java.lang.Object.clone(Native Method)
java.util.Arrays.sort(Arrays.java:1206)
java.util.Collections.sort(Collections.java:157)
IceInternal.RoutableReference.createConnection(RoutableReference.java:413)
IceInternal.DirectReference.getConnection(DirectReference.java:199)
Ice._ObjectDelM.setup(_ObjectDelM.java:258)
Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1050)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:125)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)
com.dec,dd.AgentManager.testAutoServerConnection(AgentManager.java:177)

4/10 snapshots sharing following 15 elements
java.lang.Throwable.fillInStackTrace(Native Method)
java.lang.Throwable.(Throwable.java:196)
java.lang.Exception.(Exception.java:41)
java.lang.RuntimeException.(RuntimeException.java:43)
java.lang.ClassCastException.(ClassCastException.java:39)
Ice.ObjectAdapterI.isLocal(ObjectAdapterI.java:576)
IceInternal.ObjectAdapterFactory.findObjectAdapter(ObjectAdapterFactory.java:193)
Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1038)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:125)
Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)
com.dec,dd.AgentManager.testAutoServerConnection(AgentManager.java:177)

we think there is a ConnectionI in wrong state ,it 's _ exception field is not null , but state is not close,

is there anyone encounter the problem or is it known bug , please help, thank you !

Tagged:

Answers

  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff

    Hi,

    Note that we don't provide free support for such old Ice versions, the best would be to upgrade to the latest Ice version. In any case, I don't see why Ice would indefinitely loop here. Can you verify that it's not your code which is actually looping calling ice_ping? You could for example add some tracing in the implementation of dd.AgentManager.testAutoServerConnection.

    Cheers,
    Benoit.

  • NinaNina Nina ChanOrganization: PersonalProject: monitor potential exception or problem for busniess system and send alertMember

    hi, benoit
    we have checked our code , we don't loop calling ice_ping in our code ,we found there is indefinite loop in ice_ping method ,as follows :
    private void ice_ping(Map __context, boolean __explicitCtx)
    {
    if ((__explicitCtx) && (__context == null))
    {
    __context = _emptyContext;
    }

    int __cnt = 0;
    for (;;)
    {
    _ObjectDel __del = null;
    try
    {
    __del = __getDelegate();
    __del.ice_ping(__context);
    return;
    }
    catch (LocalExceptionWrapper __ex)
    {
    __cnt = __handleExceptionWrapperRelaxed(__del, __ex, __cnt);
    }
    catch (LocalException __ex)
    {
    __cnt = __handleException(__del, __ex, __cnt);
    }
    }
    }

    when calling ice_ping, two exceptions throw ,one is ClassCastException thrown by findObjectAdapter method , its message is IceInternal.DirectReference cannot be cast to IceInternal.IndirectReference ,another exception is LocalExceptionWrapper, it thrown by sendRequest method, and it wraps a TimeoutException, the ice_ping method catch the ExceptionWrapper, and retry ,but the sendRequest still throws LocalExceptionWrapper, so ice_ping indefinitely loop。but we can't find out cause of the problem,more exception is:
    java.lang.Throwable.fillInStackTrace(Native Method)
    java.lang.Throwable.(Throwable.java:181)
    java.lang.Exception.(Exception.java:29)
    IceInternal.LocalExceptionWrapper.(LocalExceptionWrapper.java:16)
    Ice.ConnectionI.sendRequest(ConnectionI.java:534)
    IceInternal.Outgoing.invoke(Outgoing.java:72)
    Ice._ObjectDelM.ice_ping(_ObjectDelM.java:67)
    Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:126)
    Ice.ObjectPrxHelperBase.ice_ping(ObjectPrxHelperBase.java:102)

    could you give me some troubleshooting advice , we can't upgrade ice at present

  • benoitbenoit Rennes, FranceBenoit FoucherOrganization: ZeroC, Inc.Project: IceAdministrators, ZeroC Staff ZeroC Staff

    Can you try to enable Ice.Trace.Retry=1 and Ice.Trace.Network=2 to see if some tracing is printed out when this loop occurs?

    Cheers,
    Benoit.

Sign In or Register to comment.