Archived
Creating object adapter with ice grid
Ice.SocketException port already in use
When creating the object adapter
We use ice grid registry to do so by name
If we restart usually is fine, but seems like the registry can't ask OS for an unused port now all the time. I'm a little unclear though on why we'd only see this just in the last week
We did add servants to this host, and this host was rebooted along with the registry.
Comments
-
Hi,
Are you using "fixed" ports in the endpoint configuration of your object adapter? It looks like the port is already being used by another process.
Which operating system and Ice version are you using? There are usually utilities to find out which process allocated a given port on the machine.
Cheers,
Benoit.0 -
Benoit, we are not.
RHEL6.X and Ice 3.5.1Netstat reveals the port was not in use.
0 -
If you are not using a fixed port, the other possible cause for this error is that you are running out of ephemeral ports on the machine. Did
nestat
show a large number of TCP/IP connections (possibly in theTIME_WAIT
state)?0 -
no more than 3000. We have the standard port range accessible to the entire os:
[root@hostname ~]# sysctl -a | grep -i local_port
net.ipv4.ip_local_port_range = 32768 61000thats a whole ton to play around with.
0 -
Hi,
I'm a bit out of clues as to why this would happen. Can you copy/paste the exact error message printed out by the server? (especially, the exception raised by the server).
Is this happening for a single server or all your servers? Does it occur systematically or this a random failure? Did you check the system logs?
Note that it's not the IceGrid registry which requests/allocates the port. The IceGrid node just starts the server and the server asks the operating system to bind to a random available TCP/IP port. It's not clear why the operating system returns EADDRINUSE here.
Cheers,
Benoit.0 -
So just an update to our little nuisance.
We ended up finding out that the default port ranges are getting exhausted, which included the ice connections and the actual apps that requested sockets.
We ended up making the below kernel changes and we haven't seen the issue yet. RHEL has defaults for this for which anyone can look up.
Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15
Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1net.core.somaxconn = 4096
Thanks again for the assistance.
0 -
So just an update to our little nuisance.
We ended up finding out that the default port ranges are getting exhausted, which included the ice connections and the actual apps that requested sockets.
We ended up making the below kernel changes and we haven't seen the issue yet. RHEL has defaults for this for which anyone can look up.
Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15
Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1net.core.somaxconn = 4096
Thanks again for the assistance.
0 -
Good to hear you found a solution to your problem!
0