Archived

This forum has been archived. Please start a new discussion on GitHub.

Connection between Linux Server and Windows Client

Hello everybody,

I am facing a weird problem:
After creating a Linux server for registration, I now created an application on Windows (XP, VS2005). This application has to register on the linux server after starting. That is working. Now the Linux server pings the connected clients every second to see, if everything is okay. If an exception occurs during pinging a client, the client is removed from the list of registered clients from the server. This pinging is working until i shut down the windows application. Sometimes the linux server receives an exception and removing the client from the list works fine. But more often the ping simply seems to run into some kind of nirvana and no exception is thrown... The server application than hangs up and has to be hardly killed.
Amazingly the same mechanisms are running quite well when only using clients running under linux...

Any ideas?

Regards,
Jaleira

Comments

  • benoit
    benoit Rennes, France
    Hi Jaleira,

    There can be many reasons why the ping hangs and why it takes a long time to fail. Is the Windows firewall enabled perhaps? You could try to enable network tracing on the server (with --Ice.Trace.Network=2) to see when it hangs.

    What about having the client ping the server instead? If the server, doesn't receive a ping from a client within a given time interval, it would consider the client dead and remove it from its list of registered clients. The advantage here is that the server doesn't have to establish a connection to the client and it won't have to deal with clients being unreachable. See the demo/Ice/session example in the Ice distribution for an example.

    Cheers,
    Benoit.
  • Hello Benoit,

    the windows firewall is enabled, but I configured the firewall not to block the program.

    Network tracing gives the following exception on linux side after terminating the windows client:
    ./registryserver: warning: connection exception:
    TcpTransceiver.cpp:285: Ice::ConnectionLostException:
    connection lost: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
    local address = 141.*.*.118:53212
    remote address = 141.*.*.149:3521
    ./registryserver: warning: connection exception:
    TcpTransceiver.cpp:285: Ice::ConnectionLostException:
    connection lost: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
    local address = 141.*.*.118:10001
    remote address = 141.*.*.149:3518

    After that sending a ping never terminates.

    Cheers,
    Jaleira
  • My guess is still that a firewall somehow prevents proper notification about your client not running anymore.

    In any case, I recommend to set a timeout for the ping. This is a good idea in general. For example, if somebody would just "cut the cable" you wouldn't get an exception either, because the client host wouldn't be reachable at all, and thus couldn't send a notification to the server, meaning that the connection attempt (initiated by the ice_ping()) would just hang.

    I also recommend to run the server (which sends the pings to the client) with Ice.Trace.Network=2 to see what's going on.
  • I see...
    How can i set a timeout for the ping?
  • jaleira wrote:
    I see...
    How can i set a timeout for the ping?

    Please see the manual for details on how to use timeouts.
  • Okay. I set a timeout for the ping. See the source code below:
    try 
    {
    	cout << "---> sending ping"<<endl;
    	(*theIterator).servent->ice_timeout(100);
    	(*theIterator).servent->ice_ping();
    	cout << "received something on ping" <<endl;
    }//try
    catch(const Exception& ex)
    {
    	cerr<<"--> (ping) Cought Exception: "<< ex << endl;
    	(*theIterator).serviceList.clear();
    	cout << "--> (ping)" << (*theIterator).servent->ice_getIdentity().name << " is down!"<<endl;
    	emit serventDown(*theIterator);
    }
    

    During tracing the network and after shutting down the windows client I see, that Ice recognizes a Ice::ConnectionLostException, but my server does not catch this exception. So my question is: Why not?
    The server is still hanging up when trying to ping the client... (But not always, sometimes it's working and I catch some exception during pinging, so removing the client is working)

    Regards,
    Jaleira
  • It's not entirely clear to me what you mean. A server doesn't catch exceptions, all it can do is to log them.

    I don't know why your ping hangs. If there is a timeout, it should return eventually when the timeout expires.

    It would really help if you would trace the client and the server with Ice.Trace.Network=2, as suggested.
  • BTW, what might confuse you is that there are two connections, one from the client to the server, and another from the server to the client for the ice_ping(). If you use Ice.Warn.Connections=1, then you might get a ConnectionLostException warning for the connection from the client to the server. (The server can only warn here--there is no user code that could intercept this exception.) However, this has nothing to do with the connection from the server to the client, which is used for ice_ping().

    Again, trace both your client and server with Ice.Trace.Network=2, and show us the logs for both the client and the server for the time period when you get the hang. You can also add Ice.Trace.Protocol=1 for even more debug information. The warnings output from Ice.Warn.Connections are not sufficient to debug the problem.

    (Of course, all the above assumes that you do not use bi-directional connections, which you don't if you do not explicitly request them in your code.)
  • Hello again,

    first of all: thanks for the quick answers!
    Here are the Tracing messages on the client side:
    [ PolarisServent.exe: Protocol: received request
      message type = 0 (request)
      compression status = 0 (not compressed; do not compress response, if any)
      message size = 53
      request id = 7
      identity = Polaris-Servent
      facet =
      operation = ice_ping
      mode = 1 (nonmutating)
      context =  ]
    [ PolarisServent.exe: Protocol: sending reply
      message type = 2 (reply)
      compression status = 0 (not compressed; do not compress response, if any)
      message size = 25
      request id = 7
      reply status = 0 (ok) ]
    

    And here on the server side:
    ---> sending ping
    [ ./registryserver: Protocol: sending request
      message type = 0 (request)
      compression status = 0 (not compressed; do not compress response, if any)
      message size = 53
      request id = 8
      identity = Polaris-Servent
      facet =
      operation = ice_ping
      mode = 1 (nonmutating)
      context =  ]
    [ ./registryserver: Network: shutting down tcp connection for writing
      local address = 141.*.*.118:53536
      remote address = 141.*.*.221:1574 ]
    [ ./registryserver: Network: closing tcp connection
      local address = 141.3*.*.118:53536
      remote address = 141.*.*.221:1574 ]
    [ ./registryserver: Network: shutting down tcp connection for writing
      local address = 141.*.*.118:10001
      remote address = 141.*.*.221:1571 ]
    [ ./registryserver: Network: closing tcp connection
      local address = 141.*.*.118:10001
      remote address = 141.*.*.221:1571 ]
    

    Than it hangs up!

    By
    my server does not catch this exception
    i mean, that my application doesn't catch the exception although I have the try/catch stuff around pinging. Sorry for the misunderstanding.

    Regards,
    Jaleira
  • What is the trace for the client at the time when the hang occurs?

    Also, it's not clear to me that you don't get any exception. You should at least get a timeout exception.

    For even more debug information, you could use Ice.Trace.Protocol=1 together with Ice.Trace.Network=3. With the given information, I really have no clue what the problem is. Again, please provide complete traces for both the client and the server.

    Also, I still do not understand what exception you are seeing that the client doesn't catch? How do you see it?
  • Good morning!

    Here again the tracing protocols, this time with Ice.Trace.Protocol=1 and Ice.Trace.Network=3. First the protocol from the Linux-Server: At first you see a ping during normal application function. At last you see that I am sending a ping and simply nothing happens - the server hangs up. This is the case when shutting down the windows client.
    ---> sending ping
    [ ./registryserver: Protocol: sending request
      message type = 0 (request)
      compression status = 0 (not compressed; do not compress response, if any)
      message size = 53
      request id = 14
      identity = Polaris-Servent
      facet =
      operation = ice_ping
      mode = 1 (nonmutating)
      context =  ]
    [ ./registryserver: Network: sent 53 of 53 bytes via tcp
      local address = 141.*.*.118:53871
      remote address = 141.*.*.149:4417 ]
    [ ./registryserver: Network: received 14 of 14 bytes via tcp
      local address = 141.*.*.118:53871
      remote address = 141.*.*.149:4417 ]
    [ ./registryserver: Network: received 11 of 11 bytes via tcp
      local address = 141.*.*.118:53871
      remote address = 141.*.*.149:4417 ]
    [ ./registryserver: Protocol: received reply
      message type = 2 (reply)
      compression status = 0 (not compressed; do not compress response, if any)
      message size = 25
      request id = 14
      reply status = 0 (ok) ]
    received something on ping
    [ ./registryserver: Network: shutting down tcp connection for writing
      local address = 141.*.*.118:10001
      remote address = 141.*.*.149:4414 ]
    [ ./registryserver: Network: closing tcp connection
      local address = 141.*.*.118:10001
      remote address = 141.*.*.149:4414 ]
    [ ./registryserver: Network: shutting down tcp connection for writing
      local address = 141.*.*.118:53871
      remote address = 141.*.*.149:4417 ]
    [ ./registryserver: Network: closing tcp connection
      local address = 141.*.*.118:53871
      remote address = 141.*.*.149:4417 ]
    ---> sending ping
    [ ./registryserver: Network: trying to establish tcp connection to 141.*.*.149:4417 ]
    

    Protocol on Windows application side:
    [ PolarisServent.exe: Protocol: received request
      message type = 0 (request)
      compression status = 0 (not compressed; do not compress response, if any)
      message size = 53
      request id = 14
      identity = Polaris-Servent
      facet =
      operation = ice_ping
      mode = 1 (nonmutating)
      context =  ]
    [ PolarisServent.exe: Protocol: sending reply
      message type = 2 (reply)
      compression status = 0 (not compressed; do not compress response, if any)
      message size = 25
      request id = 14
      reply status = 0 (ok) ]
    [ PolarisServent.exe: Network: sent 25 of 25 bytes via tcp
      local address = 141.*.*.149:4417
      remote address = 141.*.*.118:53871 ]
    

    Refering to your question:
    Also, I still do not understand what exception you are seeing that the client doesn't catch? How do you see it?
    I saw it when running the server with Ice.Warn.Connections=1. The protocol shows a "ConnectionLostException" - i thought that this exception must be thrown and wondered why my application doesn't catch it(see again the source-code in my fourth message in this thread).

    Awaiting your advices,
    Jaleira
  • benoit
    benoit Rennes, France
    Hi,
    jaleira wrote:
    ---> sending ping
    [ ./registryserver: Network: trying to establish tcp connection to 141.*.*.149:4417 ]
    

    These traces are different from the ones above, here it's clear that the server is hanging trying to establish the connection to your Windows machine. Most likely, this is caused by the server not listening anymore on the TCP/IP port and your Windows machine ignoring the connection attempt because of the firewall. Did you try disabling the firewall to see if this makes a difference?

    If you have just "unblocked" the client executable in the Windows firewall (or added it to the list of "exceptions"), when the executable isn't running the firewall will discard connection attempts on the port. You can easily try it out by trying to connect to the IP port number with telnet ("telnet 141.*.*.149 4417"): it will also hang. However, if instead you add a port to the exceptions list of your Windows firewall configuration, Windows will reject connection attempts on the port if no programs are listening on the port.

    So if you need to use the Windows firewall, I would recommend to configure the firewall to allow the port number of your client -- this way the connection attempts will be rejected by Windows if your client isn't running anymore.
    jaleira wrote:
    Refering to your question:

    I saw it when running the server with Ice.Warn.Connections=1. The protocol shows a "ConnectionLostException" - i thought that this exception must be thrown and wondered why my application doesn't catch it(see again the source-code in my fourth message in this thread).

    No, you won't necessarily get the Ice::ConnectionLostException. Whether or not this exception is propagated to the client depends on many things.

    For example, the Ice runtime has a retry mechanism and if it's safe to retry an operation, Ice will automatically try to re-establish the connection (you can see when Ice retries or not by enabling retry tracing with --Ice.Trace.Retry=2).

    This connection warning message might also be for a different connection than the connection from your server to your client: it could be for the connection from your client to the server for instance.

    In any case, if you set a timeout on the proxy (or use Ice.Override.Timeout), the ping call which is hanging should eventually throw an Ice::TimeoutException.

    Cheers,
    Benoit.
  • benoit wrote:
    In any case, if you set a timeout on the proxy (or use Ice.Override.Timeout), the ping call which is hanging should eventually throw an Ice::TimeoutException.

    What do you mean with "eventually throw"? Shouldn't it throw something?

    Shutting down the firewall, alternatively opening the port works, but an exception thrown would be the best solution! What if somebody cut the cable and this is why the connection is lost? The ping call will never throw an exception... Now i only cover the situation that the clilent shuts down and is behind a firewall.

    Regards,
    Jaleira
  • benoit
    benoit Rennes, France
    Hi,

    Sorry, I meant that it should definitely throw after the timeout expires. If you set a timeout and your server doesn't throw an exception after the timeout expires then something is wrong. How do you set the timeout?

    I just tried with the hello world demo (located in the demo/Ice/hello directory of your Ice distribution) in a similar configuration (hello client running on a Mac OS X machine and the hello server running on Windows XP with the Windows firewall enabled and the server "unblocked") and it works as expected for me. If I shutdown the server and don't use the timeout in the client, the client hangs trying to re-establish the connection. If use the timeout (the "T" option in the hello client), I get an Ice::ConnectionRefusedException after the timeout expires.

    Could your perhaps try the demo to see if you can reproduce the issue you're getting with your client and server? You should enable the network and retry tracing on the client to see what Ice is doing (--Ice.Trace.Network=2 --Ice.Trace.Retry=2).

    Cheers,
    Benoit.
  • Since all this is happening when you shut down the windows client, I believe my original assessment above is correct. The client closes the connection that the server has opened for callbacks upon shutdown. Subsequent pings from the server to the client do not succeed in opening a new connection. The firewall blocks any notification that the client is not running anymore, and therefore, if you do not use timeouts, the ping hangs. As for why you do not get a timeout, I do not now. You definitely should (provided that you use them properly), as this has been tested in hundreds of applications, so it would surprise me very much if there would be a bug with our timeouts.
  • Okay, I got the demo running (server on windows, client on linux).
    This I get for the client:
    [ ./IceDemo: Network: trying to establish tcp connection to 141.*.*.149:10000 ]
    [ ./IceDemo: Network: tcp connection established
      local address = 141.*.*.118:54117
      remote address = 141.*.*.149:10000 ]
    usage:
    t: send greeting as twoway
    o: send greeting as oneway
    O: send greeting as batch oneway
    d: send greeting as datagram
    D: send greeting as batch datagram
    f: flush all batch requests
    T: set a timeout
    S: switch secure mode on/off
    s: shutdown server
    x: exit
    ?: help
    ==> t
    ==> [ ./IceDemo: Network: shutting down tcp connection for writing
      local address = 141.*.*.118:54117
      remote address = 141.*.*.149:10000 ]
    [ ./IceDemo: Network: closing tcp connection
      local address = 141.*.*.118:54117
      remote address = 141.*.*.149:10000 ]
    
    t
    [ ./IceDemo: Network: trying to establish tcp connection to 141.3.81.149:10000 ]
    [ ./IceDemo: Retry: connection to endpoint failed and no more endpoints to try
      Network.cpp:675: Ice::ConnectFailedException:
      connect failed: Connection timed out ]
    [ ./IceDemo: Retry: retrying operation call because of exception
      Network.cpp:675: Ice::ConnectFailedException:
      connect failed: Connection timed out ]
    [ ./IceDemo: Retry: cannot retry operation call because retry limit has been exceeded
      ConnectionFactory.cpp:103: Ice::CommunicatorDestroyedException:
      communicator object destroyed ]
    ConnectionFactory.cpp:103: Ice::CommunicatorDestroyedException:
    communicator object destroyed
    ==>   
    

    This for the server:
    [ server: Network: attempting to bind to tcp socket 141.*.*.149:10000 ]
    [ server: Network: accepting tcp connections at 141.*.*.149:10000 ]
    [ server: Network: attempting to bind to tcp socket 127.0.0.1:10000 ]
    [ server: Network: accepting tcp connections at 127.0.0.1:10000 ]
    [ server: Network: attempting to bind to udp socket 141.*.*.149:10000 ]
    [ server: Network: starting to receive udp packets
      local address = 141.*.*.149:10000
      remote address = <not connected> ]
    [ server: Network: attempting to bind to udp socket 127.0.0.1:10000 ]
    [ server: Network: starting to receive udp packets
      local address = 127.0.0.1:10000
      remote address = <not connected> ]
    [ server: Network: accepted tcp connection
      local address = 141.*.*.149:10000
      remote address = 141.*.*.118:54117 ]
    Hello World!
    [ server: Network: stopping to accept tcp connections at 141.*.*.149:10000 ]
    [ server: Network: stopping to accept tcp connections at 127.0.0.1:10000 ]
    [ server: Network: closing udp connection
      local address = 141.*.*.149:10000
      remote address = <not connected> ]
    [ server: Network: closing udp connection
      local address = 127.0.0.1:10000
      remote address = <not connected> ]
    [ server: Network: shutting down tcp connection for writing
      local address = 141.*.*.149:10000
      remote address = 141.*.*.118:54117 ]
    [ server: Network: closing tcp connection
      local address = 141.*.*.149:10000
      remote address = 141.*.*.118:54117 ]
    

    Before the timeout on client side occurs it takes nearly a minute... So have to check my source code, but i thought that a catch(const Exception& ex) catches every exception thrown by ice, doesn't it? Anyway the code fragment doing the ping is the following:
    servent->ice_timeout(100);
    servent->ice_ping();
    

    Cheers,
    Jaleira
  • jaleira wrote:
    Before the timeout on client side occurs it takes nearly a minute... So have to check my source code, but i thought that a catch(const Exception& ex) catches every exception thrown by ice, doesn't it? Anyway the code fragment doing the ping is the following:
    servent->ice_timeout(100);
    servent->ice_ping();
    

    Oops... I didn't catch this before. The code above is wrong. Proxies are immutable. You can only create a new proxy from an existing proxy. Therefore the code should be:
    servent = servent->ice_timeout(100);
    servent->ice_ping();
    

    And yes, Ice::Exception catches all Ice exceptions (user and local).