Archived

This forum has been archived. Please start a new discussion on GitHub.

a growing number of connections

loktevoleg
loktevoleg
in Help Center

I am maintaining a c++11 application with Ice 3.7.3 version on linux. It is a service for caching a database. It continiously makes requests to remote DB and stores it in memory. There is another proxy service between this one and the database which makes connection and sends requests to DB. Ice is between these two services. I use tcp to connect them. For some scale reasons I decided to put the proxy service into several docker containers and balance requests between them by haproxy (roundrobin algorythm). But when I tried it (1 cache to 2 proxies) my service started to produce a lot of connections to proxy services. The connections are being established successfully but the amount of it keeps growng with every request and the work is spoiled. There was no problem with this when I tried to connect to only one proxy service and there was no problem when I set the ip/port of proxy as an endpoint in my config directly. The containers with proxies have the same ip/port to have an ability to balance requests. So if I set this one in the config, I have this issue. Also I've tried to play with timeouts and it didn't help. I'm sure that the reconnections appear much more often then it may become old (many times per second). I don't see the huge amount of connections in haproxy. So it looks like I have something wrong in my ice part which I don't understand well enough. I'm not sure if something rips up the connection because I see no exceptions telling about it.

Tagged:

Comments

  • xdm
    xdm La Coruña, Spain

    Hi Oleg,

    Did you try enabling Ice network tracing to figure out why the connections are being open?

    https://doc.zeroc.com/ice/3.7/property-reference/ice-trace#id-.Ice.Trace.*v3.7-Ice.Trace.Network

    Setting Ice.Trace.Network=2 could give you some clues why the connections are being open.

  • loktevoleg
    loktevoleg
    edited January 2022

    Hi, Jose, thanks for your reply!
    I've tried to set Ice.Trace.Network=3. I can see authorization method being called every time and it finishes successfully. Also I see that some small data pieces are being sent and received via tcp but still it doesn't help me to understand what happens. Also I noticed that sometimes, rarely everything works fine even with 2 containers. But I didn't get when exactly. I tried to stop and run all the containers many times in different order and it looks like random. Sometimes at the first time it works fine and then after client (cache) container rerun it starts producing connections
    [edit] Now I try to make work with scheme like this: One client container + two proxy containers. One of the proxy containers produces connections and the other one does not get requests, but I can see some small data being transmitted via tcp there. It looks like several pieces being sent and received in round (like a handshake probably).
    I also tried to add the third proxy container. In this case the first one processes requests normally, the second producess connections and the third looks silent. So there is one more "role" appeared here - the one who can process requests well while others fail

  • xdm
    xdm La Coruña, Spain

    Hi,

    I've tried to set Ice.Trace.Network=3

    The tracing should also show the closing and open of the connections

    If I understand your setup correctly you are using HAProxy to load balance your requests to the (proxy containers), a small diagram of your setup with the IP address will make things clear and help understand your scenario.

    You previously wrote "The containers with proxies have the same ip/port to have an ability to balance requests", this is a bit confusing, in a well configured setup each server has to have an unique address ip/port.

    You should also review Connection Management to understand how Ice reuse connections https://doc.zeroc.com/ice/3.7/client-server-features/connection-management

    I can see some small data being transmitted via tcp there.

    These are probably ACM keep alive messages, read connection management mentioned above.

  • loktevoleg
    loktevoleg

    You are correct about my setup. To simplify it I can say that the problem can be reproduced when 1 "client" and 2 "proxies" are based on the same machine.
    I mean that the "client" container doesn't know about the amount of "proxy" ones. HAProxy meets it in one endpoint and then dispatches requests on several "proxies", each with it's own port (but same ip).

  • xdm
    xdm La Coruña, Spain 
                            ---- Ice Proxy Server 1 ---- 
                        |                          |
client ---- HAProxy --- |                          | ----- Cache Server
                        |                          |
                        ---- Ice Proxy Server 2 ---|

    Just to make sure I understand your problem, you have something like above and the number of connections between (Ice Proxy Server 2) and the Cache Server keep growing?

  • loktevoleg
    loktevoleg

    The number of connections between one of Ice Proxy Servers and client keeps growing

  • xdm
    xdm La Coruña, Spain

    Here the client is supposed to only connect to HAProxy, there shouldn't be any connections between the client and the Ice Proxy Servers.

    Can you enable network an protocol tracing in both the client and the servers and attach the logs?

    Would be also interesting to see the client and the HAProxy configurations.

    Cheers,
    Jose

  • loktevoleg
    loktevoleg

    I found a bug in my code. There was an excpetion which leaded to the endless cycle

  • xdm
    xdm La Coruña, Spain

    Hi Oleg,

    I'm glad you figured this out. Can you update your profile to describe the project you are working on.

    Cheers,
    Jose

Categories