Archived
This forum has been archived. Please start a new discussion on GitHub.
Periodic ObjectNotExistException with Glacier2 categories
joshmoore
Germany
in Bug Reports
We are seeing intermittent failures when sending stringified callback proxies to the server. Glacier2 is producing categories that end in a single "\" which leads to a category-less server-side identity when calling stringToProxy.
See https://github.com/joshmoore/zeroc-ice/commit/af0f3dd7082a397d903df1464b94ce53d989bbe5 for modifications to "java/demo/Ice/callback" which reproduce this issue. After compiling, run "java Server" to start listening. "java Client" will create callbacks in a loop using the mkident method which simulates the Glacier2 code.
After a number of iterations, the server will fail with an ObjectNotExistException. E.g:
The frequency of this happening is only a little over 1%:
so it's taken us some time to track it down (~2 years). See https://trac.openmicroscopy.org.uk/ome/ticket/8266.
Suggested workarounds would be appreciated. Attempts to modify the server-side stringToProxy code weren't successful. For the moment, we're detecting that a client-category ends in "\" and are closing and re-opening the session.
Cheers,
~Josh
See https://github.com/joshmoore/zeroc-ice/commit/af0f3dd7082a397d903df1464b94ce53d989bbe5 for modifications to "java/demo/Ice/callback" which reproduce this issue. After compiling, run "java Server" to start listening. "java Client" will create callbacks in a loop using the mkident method which simulates the Glacier2 code.
After a number of iterations, the server will fail with an ObjectNotExistException. E.g:
... initiating callback:Y4i3gZcg$azF4*}93[<X/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324 initiating callback:BC}`V?a\"ab;aj[Cxd(VC/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324 initiating callback:,X2QNUAzSBcJ_e$AV;E\\/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324 Ice.ObjectNotExistException id.name = ",X2QNUAzSBcJ_e$AV;E\/callbackReceiver" id.category = "" facet = "" operation = "callback" at IceInternal.Outgoing.invoke(Outgoing.java:158) at Demo._CallbackReceiverDelM.callback(_CallbackReceiverDelM.java:33) at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:54) at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:29) at CallbackSenderI.initiateCallbackByString(CallbackSenderI.java:37) at Demo._CallbackSenderDisp.___initiateCallbackByString(_CallbackSenderDisp.java:107) at Demo._CallbackSenderDisp.__dispatch(_CallbackSenderDisp.java:164) at IceInternal.Incoming.invoke(Incoming.java:222) at Ice.ConnectionI.invokeAll(ConnectionI.java:2482) at Ice.ConnectionI.dispatch(ConnectionI.java:1258) at Ice.ConnectionI.message(ConnectionI.java:1213) at IceInternal.ThreadPool.run(ThreadPool.java:321) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693) at java.lang.Thread.run(Thread.java:744)
The frequency of this happening is only a little over 1%:
$ cat freq.py import random import Ice import Glacier2 def mkident(): ident = Ice.Identity() ident.category = "" ident.name = "processor" size=20 for x in range(size): c = random.randint(0, 255) ident.category += chr(33+c%(127-33)) return ident found=0 size=100000 for x in range(size): c = mkident().category if c[-1] == "\\" and c[-2] != "\\": found += 1 print "%2.4f%%" % (float(found)/size * 100) $ python freq.py 1.1820%
so it's taken us some time to track it down (~2 years). See https://trac.openmicroscopy.org.uk/ome/ticket/8266.
Suggested workarounds would be appreciated. Attempts to modify the server-side stringToProxy code weren't successful. For the moment, we're detecting that a client-category ends in "\" and are closing and re-opening the session.
Cheers,
~Josh
0
Comments
-
Hi,
Thanks for the bug report, we'll look into fixing this.
Cheers,
Benoit.0 -
Hi Josh,
It's actually a bug in stringToIdentity, which does not handle properly trailing escaped escapes in the category. Thanks again for reporting it! We will include a fix in the next Ice release.
For now, you could:
1) rework your code to avoid using an intermediary stringified representation of these proxies - in general, I would recommend not to stringify transient proxies such as these Glacier2 callback proxies.
or
2) fix stringToIdentity in all language mappings where you perform these "un-stringifications" with stringToProxy, stringToIdentity, propertyToProxy.
For example, in Java, replace line 243 in src/Ice/Util.java (Ice 3.5.1)if(pos == 0 || s.charAt(pos - 1) != '\\')
withint escapes = 0; while(pos - escapes > 0 && s.charAt(pos - escapes - 1) == '\\') { escapes++; } // // We ignore escaped escapes // if(escapes % 2 == 0)
or
3) Change the Glacier2 random-category generation code to avoid generating these trailing backslashes (or backslashes in general).
Best regards,
Bernard0