Hi Jean Christophe,
Did you try replacing Ice::Service with Ice::Application in your test app? That's really the only major difference I can think of between IceBox and your test app.
In any case, it sounds like your making good progress to get to the bottom of this. The easiest way to figure out the problem would be to get a stack dump of the hang, for this you need the PDBs of the release 3rd party DLL.
Also note that IceBox isn't a requirement for running Ice services on IceGrid... you could also very well just implement your service as a regular executable linked with the 3rd party DLL if this issue can't easily be worked around.
I was looking at this definitions in https://github.com/ome/omero-blitz/blob/718d598a49c0a5db1e0a7752200d07f1ef9b472a/src/main/slice/omero/Tables.ice#L130
My understanding is that the error is not related to the rowNumbers param but with the marshaling for the Data::columns, can you dump the Data::colums to check if it has the expected values. I assuming here you are marshaling StringColumn and it contains and unexpected (not string) value.
Since you're in the process of upgrading, I strongly recommend to upgrade directly to Ice 3.7 instead of 3.6.
You should definitely be able to get a 120s invocation timeout. What is the exception you are getting in scenario #4? Is it an Ice::InvocationTimeoutException or an Ice::ConnectiontTimeoutException? Are both the client and server using Ice 3.6 or do you still use Ice 3.5 for the server?
Invocation timeouts rely on the server sending "heartbeats" while the long running invocation is being dispatch. If the server is still running 3.5, it won't send heartbeats and the Ice 3.6 client will mistakenly assume that the server is gone, the default 60s connection timeout will therefore kick in.
See the following link for additional information on upgrading timeouts from 3.5: https://doc.zeroc.com/ice/3.7/release-notes/upgrading-your-application-from-ice-3-5#id-.UpgradingyourApplicationfromIce3.5v3.7-Timeoutchanges