Home Bug Reports

Periodic ObjectNotExistException with Glacier2 categories

joshmoorejoshmoore GermanyMember Josh MooreOrganization: Glencoe Software, Inc.Project: OME, http://openmicroscopy.org
We are seeing intermittent failures when sending stringified callback proxies to the server. Glacier2 is producing categories that end in a single "\" which leads to a category-less server-side identity when calling stringToProxy.

See https://github.com/joshmoore/zeroc-ice/commit/af0f3dd7082a397d903df1464b94ce53d989bbe5 for modifications to "java/demo/Ice/callback" which reproduce this issue. After compiling, run "java Server" to start listening. "java Client" will create callbacks in a loop using the mkident method which simulates the Glacier2 code.

After a number of iterations, the server will fail with an ObjectNotExistException. E.g:
...
initiating callback:Y4i3gZcg$azF4*}93[<X/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324
initiating callback:BC}`V?a\"ab;aj[Cxd(VC/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324
initiating callback:,X2QNUAzSBcJ_e$AV;E\\/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324
Ice.ObjectNotExistException
    id.name = ",X2QNUAzSBcJ_e$AV;E\/callbackReceiver"
    id.category = ""
    facet = ""
    operation = "callback"
        at IceInternal.Outgoing.invoke(Outgoing.java:158)
        at Demo._CallbackReceiverDelM.callback(_CallbackReceiverDelM.java:33)
        at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:54)
        at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:29)
        at CallbackSenderI.initiateCallbackByString(CallbackSenderI.java:37)
        at Demo._CallbackSenderDisp.___initiateCallbackByString(_CallbackSenderDisp.java:107)
        at Demo._CallbackSenderDisp.__dispatch(_CallbackSenderDisp.java:164)
        at IceInternal.Incoming.invoke(Incoming.java:222)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
        at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
        at Ice.ConnectionI.message(ConnectionI.java:1213)
        at IceInternal.ThreadPool.run(ThreadPool.java:321)
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
        at java.lang.Thread.run(Thread.java:744)


The frequency of this happening is only a little over 1%:
$ cat freq.py
import random
import Ice
import Glacier2

def mkident():
    ident = Ice.Identity()
    ident.category = ""
    ident.name = "processor"
    size=20
    for x in range(size):
        c = random.randint(0, 255)
        ident.category += chr(33+c%(127-33))
    return ident

found=0
size=100000
for x in range(size):
    c = mkident().category
    if c[-1] == "\\" and c[-2] != "\\":
        found += 1
print "%2.4f%%" % (float(found)/size * 100)

$ python freq.py 
1.1820%

so it's taken us some time to track it down (~2 years). See https://trac.openmicroscopy.org.uk/ome/ticket/8266.

Suggested workarounds would be appreciated. Attempts to modify the server-side stringToProxy code weren't successful. For the moment, we're detecting that a client-category ends in "\" and are closing and re-opening the session.

Cheers,
~Josh

Comments

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi,

    Thanks for the bug report, we'll look into fixing this.

    Cheers,
    Benoit.
  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi Josh,

    It's actually a bug in stringToIdentity, which does not handle properly trailing escaped escapes in the category. Thanks again for reporting it! We will include a fix in the next Ice release.

    For now, you could:

    1) rework your code to avoid using an intermediary stringified representation of these proxies - in general, I would recommend not to stringify transient proxies such as these Glacier2 callback proxies.

    or
    2) fix stringToIdentity in all language mappings where you perform these "un-stringifications" with stringToProxy, stringToIdentity, propertyToProxy.

    For example, in Java, replace line 243 in src/Ice/Util.java (Ice 3.5.1)
                if(pos == 0 || s.charAt(pos - 1) != '\\')
    
    with
                int escapes = 0;
                while(pos - escapes > 0 && s.charAt(pos - escapes - 1) == '\\')
                {
                    escapes++;
                }
                   
                //
                // We ignore escaped escapes
                //
                if(escapes % 2 == 0)
    

    or
    3) Change the Glacier2 random-category generation code to avoid generating these trailing backslashes (or backslashes in general).

    Best regards,
    Bernard
Sign In or Register to comment.