Syscall exception on Win2K using IcePack.Node.Output

brian

We are seeing another weird OS-dependent bug and have some data points to relate to you after some extensive
investigation.


We have started using the IcePack.Node.Output property, but have run into a problem on Windows 2K only,
the problem does NOT exist on Solaris. We are using v1.4. CC compiler version is 5.3 Patch 111685-20 2004/03/19.

What happens is this: we have two icebox services running on a node on a Win2K machine, everything
else is running on Solaris. One of the services, "RobotradeServer" referenced in the error message below,
is explicitly started and stopped using the IcePack::Admin interface. The other service, call it "Foo" is started/stopped
automatically by IcePack. There is one servant maintained by each service. This sequence of actions causes the problem:

1) start both services
2) stop both services
3) restart both services

icepacknode spits this out:


icepacknode.exe: warning: activation failed for server `RobotradeServer':
.\ActivatorI.cpp:559: Ice::SyscallException:
syscall exception: The process cannot access the file because it is being used by another process.

icepacknode.exe: warning: activation failed for server `RobotradeServer':
.\ActivatorI.cpp:559: Ice::SyscallException:
syscall exception: The process cannot access the file because it is being used by another process.

icepacknode.exe: warning: activation failed for server `RobotradeServer':
.\ActivatorI.cpp:559: Ice::SyscallException:
syscall exception: The process cannot access the file because it is being used by another process.



This problem does NOT happen if:

1) We do not use IcePack.Node.Output
2) we move at least one of the two services from Win2K to Solaris

The four files generated for logging on Win2K are RobotradeServer.out and .err and Foo.out and Foo.err.

Any ideas?

Thanks again,

Brian

brian

Hi Guys,

Just wondering if anyone has had a chance to look into this problem. If it is something somebody is investigating that is fine. Was hopeful this issue was at least on the radar at this point.

Always appreciate your help.

Brian

bernard

Hi Brian,

Sorry for the late reply! I tried to reproduce this problem using the IcePack/simple demo, but could not. However I got immediately a:

C:\builds\vc60\ice\src\IcePack\Activator.cpp:564: Ice::SyscallException:
syscall exception: The process cannot access the file because it is being used by another process.

after commenting out the code that closes the stdout file in Activator.cpp (was ActivatorI.cpp in Ice 1.4).

This cleanup code runs when IcePackNode detects server termination, so one possibility is that somehow the termination of your server is not properly detected. Could you enable tracing in your IcePackNode with
IcePack.Node.Trace.Activator=3 to check if that's indeed the issue?

Cheers,
Bernard

brian

Bernard,

Thanks for the reply. I have turned on that tracing you suggested and am doing some further investigation. Will get back to you sometime next week.


Thanks,

Brian

brian

Bernard,

I have some results of the trace for you in included in the attached file.

Thanks,

Brian

bernard

Hi Brian,

Everything looks ok in this log. However I still can't reproduce this problem. I tried with the Ice 1.4.0 IcePack/simple demo: it also has two servers, and I can easily start/stop one with icepackadmin, and print lines on either server stdout with 'icepackadmin server stdout'.

Maybe it's because RobotradeServer talks to your other server (unlike the servers in IcePack/simple) and hangs at exit. Could you replace your src/Ice/Network.cpp by the attached file, rebuild and try again? Or (if you prefer) upgrade to Ice 1.5.1, which also includes this fix. This may solve your problem.

Cheers,
Bernard

brian

Hi Bernard,

I tried the new Network.cpp file, but it did not fix the problem.

If you look at the log, when we first stop the servers, only "RobotradeServer" says it is being deactivated, not "Demo_Validator". Could this be a clue?

Brian

brian

Bernard,

Upon further investigation, when I commented out the explicit call to IcePack::Admin::stopServer, the syscall exception stopped. I still call startServer each time I start the service, though.

Any ideas on why this stopped the problem and what would be the implications of not calling stopServer explicitly when I want to stop the this (RobotradeServer) service that is associated with another servant?

Thanks,


Brian

bernard

Hi Brian,

How do you deactivate the services when you don't use an explicit "stop" call?
Does RobotTradeServer call Demo_Validator or vice-versa?

Are your servers implemented in C++ or Java? The work-around above is just for C++ servers. The current implementation of IcePackNode.Output on Windows isn't good ... we will rewrite it properly for the next Ice release. On Windows, the safest is to:
- upgrade to Ice 1.5.1
- not use the IcePackNode.Ouput feature

(IcePackNode.Output works properly on Linux/Unix in Ice 1.4/1.5).

Cheers,
Bernard

brian

Bernard,


How do you deactivate the services when you don't use an explicit "stop" call?

We call deactivate on the adapter that contains the object.

Does RobotTradeServer call Demo_Validator or vice-versa?

No.


Are your servers implemented in C++ or Java?

Both. Demo_Validator and RobotradeServer are Java. The service that explicitly starts/stops RobotradeServer using the Admin API is in C++.

Don't know if you read my previous message that indicates that I tried your Network.cpp and it did not fix the problem. Is it still worth it to take the time to upgrade from 1.4 if this did not work?

Thanks,

Brian

bernard

On Windows, IcePackNode.Output is currently implemented by opening files for stdout and stderr in the parent process (IcePackNode) and passing them to the child. We open these files in IcePackNode to implement the server stdout and server stderr icepackadmin commands.

This handle-passing requires to set the "Inherit handles" paramater of the Win32 CreateProcess call to TRUE. Unfortunately other handles are inherited as well, in particular sockets (network connections established by IcePackNode).

Inheriting these sockets results in problems tearing down the connections with IcePackNode ... client hangs. The work-around in 1.5.1 and the Network.cpp in my previous message is to make sockets non-inheritable after creation. This is just a work-around because we can have race conditions: a new process being created while a socket is created but not yet marked non-inheritable. In 1.5.1, if you don't set IcePackNode.Output, we pass FALSE to CreateProcess (do not inherit handles and avoid this issue altogether).

This updated Network.cpp applies to the IcePackNode process itself, and (unlike what I mistakenly wrote earlier) works for servers written in any language.

Now, I need to try the Java version of the IcePack/simple demo to see if I can reproduce your problem.

> If you look at the log, when we first stop the servers, only "RobotradeServer" says it is being deactivated, not "Demo_Validator". Could this be a clue?

This is strange, maybe a different problem? I don't see how Demo_Validator could lock RobotradeServer's files.

>Don't know if you read my previous message that indicates that I tried your Network.cpp and it did not fix the problem. Is it still worth it to take the time to upgrade from 1.4 if this did not work?

So far I doubt it would make any difference.

Cheers,
Bernard

bernard

I was able to reproduce the problem immediately with Ice for Java ... I don't know why, maybe luck.
The second started server inherits the file handles of the first server, and prevents the first server from restarting. You'll find attached a patched src/IcePack/ActivatorI.cpp (for Ice 1.4.0) that works around this problem.

Let me know how it goes!

Cheers,
Bernard

brian

Bernard,

Don't have the capability to compile source on Windows. Any chance you could apply the patch and send me an updated icePack14.dll and icepacknode?

Brian

bernard

Hi Brian,

Which compiler do you use? And could you e-mail me a valid e-mail address?

Cheers,
Bernard

bernard@zeroc.com