Syscall exception on Win2K using IcePack.Node.Output

brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
We are seeing another weird OS-dependent bug and have some data points to relate to you after some extensive
investigation.


We have started using the IcePack.Node.Output property, but have run into a problem on Windows 2K only,
the problem does NOT exist on Solaris. We are using v1.4. CC compiler version is 5.3 Patch 111685-20 2004/03/19.

What happens is this: we have two icebox services running on a node on a Win2K machine, everything
else is running on Solaris. One of the services, "RobotradeServer" referenced in the error message below,
is explicitly started and stopped using the IcePack::Admin interface. The other service, call it "Foo" is started/stopped
automatically by IcePack. There is one servant maintained by each service. This sequence of actions causes the problem:

1) start both services
2) stop both services
3) restart both services

icepacknode spits this out:


icepacknode.exe: warning: activation failed for server `RobotradeServer':
.\ActivatorI.cpp:559: Ice::SyscallException:
syscall exception: The process cannot access the file because it is being used by another process.

icepacknode.exe: warning: activation failed for server `RobotradeServer':
.\ActivatorI.cpp:559: Ice::SyscallException:
syscall exception: The process cannot access the file because it is being used by another process.

icepacknode.exe: warning: activation failed for server `RobotradeServer':
.\ActivatorI.cpp:559: Ice::SyscallException:
syscall exception: The process cannot access the file because it is being used by another process.



This problem does NOT happen if:

1) We do not use IcePack.Node.Output
2) we move at least one of the two services from Win2K to Solaris

The four files generated for logging on Win2K are RobotradeServer.out and .err and Foo.out and Foo.err.

Any ideas?

Thanks again,

Brian

Comments

  • brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
    Hi Guys,

    Just wondering if anyone has had a chance to look into this problem. If it is something somebody is investigating that is fine. Was hopeful this issue was at least on the radar at this point.

    Always appreciate your help.

    Brian
  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi Brian,

    Sorry for the late reply! I tried to reproduce this problem using the IcePack/simple demo, but could not. However I got immediately a:

    C:\builds\vc60\ice\src\IcePack\Activator.cpp:564: Ice::SyscallException:
    syscall exception: The process cannot access the file because it is being used by another process.

    after commenting out the code that closes the stdout file in Activator.cpp (was ActivatorI.cpp in Ice 1.4).

    This cleanup code runs when IcePackNode detects server termination, so one possibility is that somehow the termination of your server is not properly detected. Could you enable tracing in your IcePackNode with
    IcePack.Node.Trace.Activator=3 to check if that's indeed the issue?

    Cheers,
    Bernard
  • brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
    Bernard,

    Thanks for the reply. I have turned on that tracing you suggested and am doing some further investigation. Will get back to you sometime next week.


    Thanks,

    Brian
  • brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
    Bernard,

    I have some results of the trace for you in included in the attached file.

    Thanks,

    Brian
  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi Brian,

    Everything looks ok in this log. However I still can't reproduce this problem. I tried with the Ice 1.4.0 IcePack/simple demo: it also has two servers, and I can easily start/stop one with icepackadmin, and print lines on either server stdout with 'icepackadmin server stdout'.

    Maybe it's because RobotradeServer talks to your other server (unlike the servers in IcePack/simple) and hangs at exit. Could you replace your src/Ice/Network.cpp by the attached file, rebuild and try again? Or (if you prefer) upgrade to Ice 1.5.1, which also includes this fix. This may solve your problem.

    Cheers,
    Bernard
  • brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
    Hi Bernard,

    I tried the new Network.cpp file, but it did not fix the problem.

    If you look at the log, when we first stop the servers, only "RobotradeServer" says it is being deactivated, not "Demo_Validator". Could this be a clue?

    Brian
  • brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
    Bernard,

    Upon further investigation, when I commented out the explicit call to IcePack::Admin::stopServer, the syscall exception stopped. I still call startServer each time I start the service, though.

    Any ideas on why this stopped the problem and what would be the implications of not calling stopServer explicitly when I want to stop the this (RobotradeServer) service that is associated with another servant?

    Thanks,


    Brian
  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi Brian,

    How do you deactivate the services when you don't use an explicit "stop" call?
    Does RobotTradeServer call Demo_Validator or vice-versa?

    Are your servers implemented in C++ or Java? The work-around above is just for C++ servers. The current implementation of IcePackNode.Output on Windows isn't good ... we will rewrite it properly for the next Ice release. On Windows, the safest is to:
    - upgrade to Ice 1.5.1
    - not use the IcePackNode.Ouput feature

    (IcePackNode.Output works properly on Linux/Unix in Ice 1.4/1.5).

    Cheers,
    Bernard
  • brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
    Bernard,


    How do you deactivate the services when you don't use an explicit "stop" call?

    We call deactivate on the adapter that contains the object.

    Does RobotTradeServer call Demo_Validator or vice-versa?

    No.


    Are your servers implemented in C++ or Java?

    Both. Demo_Validator and RobotradeServer are Java. The service that explicitly starts/stops RobotradeServer using the Admin API is in C++.

    Don't know if you read my previous message that indicates that I tried your Network.cpp and it did not fix the problem. Is it still worth it to take the time to upgrade from 1.4 if this did not work?

    Thanks,

    Brian
  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    On Windows, IcePackNode.Output is currently implemented by opening files for stdout and stderr in the parent process (IcePackNode) and passing them to the child. We open these files in IcePackNode to implement the server stdout and server stderr icepackadmin commands.

    This handle-passing requires to set the "Inherit handles" paramater of the Win32 CreateProcess call to TRUE. Unfortunately other handles are inherited as well, in particular sockets (network connections established by IcePackNode).

    Inheriting these sockets results in problems tearing down the connections with IcePackNode ... client hangs. The work-around in 1.5.1 and the Network.cpp in my previous message is to make sockets non-inheritable after creation. This is just a work-around because we can have race conditions: a new process being created while a socket is created but not yet marked non-inheritable. In 1.5.1, if you don't set IcePackNode.Output, we pass FALSE to CreateProcess (do not inherit handles and avoid this issue altogether).

    This updated Network.cpp applies to the IcePackNode process itself, and (unlike what I mistakenly wrote earlier) works for servers written in any language.

    Now, I need to try the Java version of the IcePack/simple demo to see if I can reproduce your problem.

    > If you look at the log, when we first stop the servers, only "RobotradeServer" says it is being deactivated, not "Demo_Validator". Could this be a clue?

    This is strange, maybe a different problem? I don't see how Demo_Validator could lock RobotradeServer's files.

    >Don't know if you read my previous message that indicates that I tried your Network.cpp and it did not fix the problem. Is it still worth it to take the time to upgrade from 1.4 if this did not work?

    So far I doubt it would make any difference.

    Cheers,
    Bernard
  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    I was able to reproduce the problem immediately with Ice for Java ... I don't know why, maybe luck.
    The second started server inherits the file handles of the first server, and prevents the first server from restarting. You'll find attached a patched src/IcePack/ActivatorI.cpp (for Ice 1.4.0) that works around this problem.

    Let me know how it goes!

    Cheers,
    Bernard
  • brianbrian Member brian whitneyOrganization: boulder research and developmentProject: MarketBank ✭✭✭
    Bernard,

    Don't have the capability to compile source on Windows. Any chance you could apply the patch and send me an updated icePack14.dll and icepacknode?

    Brian
  • bernardbernard Jupiter, FLAdministrators, ZeroC Staff Bernard NormierOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    Hi Brian,

    Which compiler do you use? And could you e-mail me a valid e-mail address?

    Cheers,
    Bernard

    [email protected]
Sign In or Register to comment.