Periodic ObjectNotExistException with Glacier2 categories

joshmoore · May 2014

We are seeing intermittent failures when sending stringified callback proxies to the server. Glacier2 is producing categories that end in a single "\" which leads to a category-less server-side identity when calling stringToProxy.

See https://github.com/joshmoore/zeroc-ice/commit/af0f3dd7082a397d903df1464b94ce53d989bbe5 for modifications to "java/demo/Ice/callback" which reproduce this issue. After compiling, run "java Server" to start listening. "java Client" will create callbacks in a loop using the mkident method which simulates the Glacier2 code.

After a number of iterations, the server will fail with an ObjectNotExistException. E.g:

...
initiating callback:Y4i3gZcg$azF4*}93[<X/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324
initiating callback:BC}`V?a\"ab;aj[Cxd(VC/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324
initiating callback:,X2QNUAzSBcJ_e$AV;E\\/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324
Ice.ObjectNotExistException
    id.name = ",X2QNUAzSBcJ_e$AV;E\/callbackReceiver"
    id.category = ""
    facet = ""
    operation = "callback"
        at IceInternal.Outgoing.invoke(Outgoing.java:158)
        at Demo._CallbackReceiverDelM.callback(_CallbackReceiverDelM.java:33)
        at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:54)
        at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:29)
        at CallbackSenderI.initiateCallbackByString(CallbackSenderI.java:37)
        at Demo._CallbackSenderDisp.___initiateCallbackByString(_CallbackSenderDisp.java:107)
        at Demo._CallbackSenderDisp.__dispatch(_CallbackSenderDisp.java:164)
        at IceInternal.Incoming.invoke(Incoming.java:222)
        at Ice.ConnectionI.invokeAll(ConnectionI.java:2482)
        at Ice.ConnectionI.dispatch(ConnectionI.java:1258)
        at Ice.ConnectionI.message(ConnectionI.java:1213)
        at IceInternal.ThreadPool.run(ThreadPool.java:321)
        at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
        at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693)
        at java.lang.Thread.run(Thread.java:744)

The frequency of this happening is only a little over 1%:

$ cat freq.py
import random
import Ice
import Glacier2

def mkident():
    ident = Ice.Identity()
    ident.category = ""
    ident.name = "processor"
    size=20
    for x in range(size):
        c = random.randint(0, 255)
        ident.category += chr(33+c%(127-33))
    return ident

found=0
size=100000
for x in range(size):
    c = mkident().category
    if c[-1] == "\\" and c[-2] != "\\":
        found += 1
print "%2.4f%%" % (float(found)/size * 100)

$ python freq.py 
1.1820%

so it's taken us some time to track it down (~2 years). See https://trac.openmicroscopy.org.uk/ome/ticket/8266.

Suggested workarounds would be appreciated. Attempts to modify the server-side stringToProxy code weren't successful. For the moment, we're detecting that a client-category ends in "\" and are closing and re-opening the session.

Cheers,
~Josh

benoit · May 2014

Hi,

Thanks for the bug report, we'll look into fixing this.

Cheers,
Benoit.

bernard · May 2014

Hi Josh,

It's actually a bug in stringToIdentity, which does not handle properly trailing escaped escapes in the category. Thanks again for reporting it! We will include a fix in the next Ice release.

For now, you could:

1) rework your code to avoid using an intermediary stringified representation of these proxies - in general, I would recommend not to stringify transient proxies such as these Glacier2 callback proxies.

or
2) fix stringToIdentity in all language mappings where you perform these "un-stringifications" with stringToProxy, stringToIdentity, propertyToProxy.

For example, in Java, replace line 243 in src/Ice/Util.java (Ice 3.5.1)

            if(pos == 0 || s.charAt(pos - 1) != '\\')

with

            int escapes = 0;
            while(pos - escapes > 0 && s.charAt(pos - escapes - 1) == '\\')
            {
                escapes++;
            }
               
            //
            // We ignore escaped escapes
            //
            if(escapes % 2 == 0)

or
3) Change the Glacier2 random-category generation code to avoid generating these trailing backslashes (or backslashes in general).

Best regards,
Bernard

Archived

Periodic ObjectNotExistException with Glacier2 categories

Comments

Categories