Creating object adapter with ice grid

tpologrutotpologruto Member Tom PologrutoOrganization: Capstone ConsultingProject: OMS
We have used Ice for many years and last week we started to randomly see this error:

Ice.SocketException port already in use

When creating the object adapter

We use ice grid registry to do so by name

If we restart usually is fine, but seems like the registry can't ask OS for an unused port now all the time. I'm a little unclear though on why we'd only see this just in the last week

We did add servants to this host, and this host was rebooted along with the registry.

Comments

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    Hi,

    Are you using "fixed" ports in the endpoint configuration of your object adapter? It looks like the port is already being used by another process.

    Which operating system and Ice version are you using? There are usually utilities to find out which process allocated a given port on the machine.

    Cheers,
    Benoit.

  • bchenbchen Member Brian ChenOrganization: Capstone Investment AdvisorsProject: internal

    Benoit, we are not.
    RHEL6.X and Ice 3.5.1

    Netstat reveals the port was not in use.

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    If you are not using a fixed port, the other possible cause for this error is that you are running out of ephemeral ports on the machine. Did nestat show a large number of TCP/IP connections (possibly in the TIME_WAIT state)?

  • bchenbchen Member Brian ChenOrganization: Capstone Investment AdvisorsProject: internal

    no more than 3000. We have the standard port range accessible to the entire os:

    [[email protected] ~]# sysctl -a | grep -i local_port
    net.ipv4.ip_local_port_range = 32768 61000

    thats a whole ton to play around with.

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    Hi,

    I'm a bit out of clues as to why this would happen. Can you copy/paste the exact error message printed out by the server? (especially, the exception raised by the server).

    Is this happening for a single server or all your servers? Does it occur systematically or this a random failure? Did you check the system logs?

    Note that it's not the IceGrid registry which requests/allocates the port. The IceGrid node just starts the server and the server asks the operating system to bind to a random available TCP/IP port. It's not clear why the operating system returns EADDRINUSE here.

    Cheers,
    Benoit.

  • bchenbchen Member Brian ChenOrganization: Capstone Investment AdvisorsProject: internal

    So just an update to our little nuisance.

    We ended up finding out that the default port ranges are getting exhausted, which included the ice connections and the actual apps that requested sockets.

    We ended up making the below kernel changes and we haven't seen the issue yet. RHEL has defaults for this for which anyone can look up.

    Decrease the time default value for tcp_fin_timeout connection

    net.ipv4.tcp_fin_timeout = 15

    Allowed local port range

    net.ipv4.ip_local_port_range = 2000 65535

    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_tw_reuse = 1

    net.core.somaxconn = 4096

    Thanks again for the assistance.

  • bchenbchen Member Brian ChenOrganization: Capstone Investment AdvisorsProject: internal

    So just an update to our little nuisance.

    We ended up finding out that the default port ranges are getting exhausted, which included the ice connections and the actual apps that requested sockets.

    We ended up making the below kernel changes and we haven't seen the issue yet. RHEL has defaults for this for which anyone can look up.

    Decrease the time default value for tcp_fin_timeout connection

    net.ipv4.tcp_fin_timeout = 15

    Allowed local port range

    net.ipv4.ip_local_port_range = 2000 65535

    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_tw_reuse = 1

    net.core.somaxconn = 4096

    Thanks again for the assistance.

  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff

    Good to hear you found a solution to your problem!

Sign In or Register to comment.