Archived
This forum has been archived. Please start a new discussion on GitHub.
Periodic ObjectNotExistException with Glacier2 categories
![joshmoore](https://forums.zeroc.com/uploads/userpics/513/n4EBVFEMC9Z3W.jpg)
joshmoore
Germany
in Bug Reports
We are seeing intermittent failures when sending stringified callback proxies to the server. Glacier2 is producing categories that end in a single "\" which leads to a category-less server-side identity when calling stringToProxy.
See https://github.com/joshmoore/zeroc-ice/commit/af0f3dd7082a397d903df1464b94ce53d989bbe5 for modifications to "java/demo/Ice/callback" which reproduce this issue. After compiling, run "java Server" to start listening. "java Client" will create callbacks in a loop using the mkident method which simulates the Glacier2 code.
After a number of iterations, the server will fail with an ObjectNotExistException. E.g:
The frequency of this happening is only a little over 1%:
so it's taken us some time to track it down (~2 years). See https://trac.openmicroscopy.org.uk/ome/ticket/8266.
Suggested workarounds would be appreciated. Attempts to modify the server-side stringToProxy code weren't successful. For the moment, we're detecting that a client-category ends in "\" and are closing and re-opening the session.
Cheers,
~Josh
See https://github.com/joshmoore/zeroc-ice/commit/af0f3dd7082a397d903df1464b94ce53d989bbe5 for modifications to "java/demo/Ice/callback" which reproduce this issue. After compiling, run "java Server" to start listening. "java Client" will create callbacks in a loop using the mkident method which simulates the Glacier2 code.
After a number of iterations, the server will fail with an ObjectNotExistException. E.g:
... initiating callback:Y4i3gZcg$azF4*}93[<X/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324 initiating callback:BC}`V?a\"ab;aj[Cxd(VC/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324 initiating callback:,X2QNUAzSBcJ_e$AV;E\\/callbackReceiver -t -e 1.1:tcp -h 172.17.42.1 -p 55324:tcp -h 192.168.178.57 -p 55324 Ice.ObjectNotExistException id.name = ",X2QNUAzSBcJ_e$AV;E\/callbackReceiver" id.category = "" facet = "" operation = "callback" at IceInternal.Outgoing.invoke(Outgoing.java:158) at Demo._CallbackReceiverDelM.callback(_CallbackReceiverDelM.java:33) at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:54) at Demo.CallbackReceiverPrxHelper.callback(CallbackReceiverPrxHelper.java:29) at CallbackSenderI.initiateCallbackByString(CallbackSenderI.java:37) at Demo._CallbackSenderDisp.___initiateCallbackByString(_CallbackSenderDisp.java:107) at Demo._CallbackSenderDisp.__dispatch(_CallbackSenderDisp.java:164) at IceInternal.Incoming.invoke(Incoming.java:222) at Ice.ConnectionI.invokeAll(ConnectionI.java:2482) at Ice.ConnectionI.dispatch(ConnectionI.java:1258) at Ice.ConnectionI.message(ConnectionI.java:1213) at IceInternal.ThreadPool.run(ThreadPool.java:321) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:693) at java.lang.Thread.run(Thread.java:744)
The frequency of this happening is only a little over 1%:
$ cat freq.py import random import Ice import Glacier2 def mkident(): ident = Ice.Identity() ident.category = "" ident.name = "processor" size=20 for x in range(size): c = random.randint(0, 255) ident.category += chr(33+c%(127-33)) return ident found=0 size=100000 for x in range(size): c = mkident().category if c[-1] == "\\" and c[-2] != "\\": found += 1 print "%2.4f%%" % (float(found)/size * 100) $ python freq.py 1.1820%
so it's taken us some time to track it down (~2 years). See https://trac.openmicroscopy.org.uk/ome/ticket/8266.
Suggested workarounds would be appreciated. Attempts to modify the server-side stringToProxy code weren't successful. For the moment, we're detecting that a client-category ends in "\" and are closing and re-opening the session.
Cheers,
~Josh
0
Comments
-
Hi,
Thanks for the bug report, we'll look into fixing this.
Cheers,
Benoit.0 -
Hi Josh,
It's actually a bug in stringToIdentity, which does not handle properly trailing escaped escapes in the category. Thanks again for reporting it! We will include a fix in the next Ice release.
For now, you could:
1) rework your code to avoid using an intermediary stringified representation of these proxies - in general, I would recommend not to stringify transient proxies such as these Glacier2 callback proxies.
or
2) fix stringToIdentity in all language mappings where you perform these "un-stringifications" with stringToProxy, stringToIdentity, propertyToProxy.
For example, in Java, replace line 243 in src/Ice/Util.java (Ice 3.5.1)if(pos == 0 || s.charAt(pos - 1) != '\\')
withint escapes = 0; while(pos - escapes > 0 && s.charAt(pos - escapes - 1) == '\\') { escapes++; } // // We ignore escaped escapes // if(escapes % 2 == 0)
or
3) Change the Glacier2 random-category generation code to avoid generating these trailing backslashes (or backslashes in general).
Best regards,
Bernard0