Ice Connection Refused Exception

aswani

Hi,

We are using the Ice Grid Locator services to connect between the Web layer and business layer.

The code is in JAVA.

The IceGrid XML for the user object is as below

<icegrid>
<application name="outduICEUser">
<node name="Node3">
<server id="outduICEUserI"
exe="java"
pwd="/home/manager/Data_Centre/apps/shared/java"
activation="on-demand">
<env>CLASSPATH=/home/manager/Data_Centre/ext/java-common/lib/commons-lang-2.4.jar:/home/manager/Data_Centre/apps/shared/java:/home/manager/Data_Centre/apps/shared/java/OutduUser.jar:./:</env>
<option>outduBI.outduICEUserIServer</option>
<adapter name="outduICEUserAdapter"
id="outduICEUserAdapter"
endpoints="tcp"/>
</server>
</node>
</application>
</icegrid>


The Locator Services works for some time. But after some time the StringToProxy() happens, but the checked cast throws a connection exception.

We are using a Virtual Machines (VMWare) with Redhat 5.1 Server.
The Webserver (tomcat) , Business Server are separate machines.
Locator is configured on the Business server.
We have only the ethernet network configured on the machine.

The Endpoints defined are absolute values (ex: tcp -h 192.168.1.153 -p 4061)

The stack trace of the exception is as shown below


Ice.ConnectionRefusedException
error = 0
at IceInternal.Network.doConnect(Network.java:320)
at IceInternal.TcpConnector.connect(TcpConnector.java:26)
at IceInternal.OutgoingConnectionFactory.create(OutgoingConnectionFactory.java:323)
at IceInternal.RoutableReference.createConnection(RoutableReference.java:433)
at IceInternal.DirectReference.getConnection(DirectReference.java:199)
at Ice._ObjectDelM.setup(_ObjectDelM.java:258)
at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1055)
at Ice.LocatorPrxHelper.findAdapterById(LocatorPrxHelper.java:46)
at Ice.LocatorPrxHelper.findAdapterById(LocatorPrxHelper.java:20)
at IceInternal.LocatorInfo.getEndpoints(LocatorInfo.java:99)
at IceInternal.IndirectReference.getConnection(IndirectReference.java:174)
at Ice._ObjectDelM.setup(_ObjectDelM.java:258)
at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1055)
at Ice.ObjectPrxHelperBase.ice_isA(ObjectPrxHelperBase.java:85)
at Ice.ObjectPrxHelperBase.ice_isA(ObjectPrxHelperBase.java:61)
at outduBI.outduICEUserIfacePrxHelper.checkedCast(outduICEUserIfacePrxHelper.java:620)
at icecomm.UserClient.<init>(UserClient.java:26)
at outduUser.OutduUser.userLogin(OutduUser.java:90)
at outduUser.OutduUser.processRequest(OutduUser.java:251)
at outduUser.OutduUser.doGet(OutduUser.java:325)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at IceInternal.Network.doConnect(Network.java:307)
... 33 more

Any help or pointers to solve the issue are appreciated

Thanks in advance

Regards,
Aswani

bernard

Hi Aswani,

This stack trace shows that your Java application is unable to establish a connection to the IceGrid registry. This could occur if the IceGrid registry is not running, or if Ice.Default.Locator is not set properly.

If the IceGrid registry is running, I suggest to run your Java application with --Ice.Trace.Network=2, to get more information on this failed connection attempt.

Cheers,
Bernard

aswani

Hi, Bernard

Some clarifications required regarding the Location Services.

If the StringToProxy() works, it means the proxy has been resolved i.e registry services worked

Now the client has to make a direct connection to server (which is done at checkedcast level)

Please point to me if am wrong

Regards,
Aswani

aswani

Hi Bernard,

We did have all the traces on at all levels (client, locator, registry, nodes).
It is not that it doesn't work at all, but it is inconsistent (it does work for some time and then doesn't respond)

Regards,
Aswani

bernard

Hi Aswani

aswani wrote: »

If the StringToProxy() works, it means the proxy has been resolved i.e registry services worked

StringToProxy is a purely local operation; in particular, it does not make a call to the location service (IceGrid registry) when it's an indirect proxy. The indirect proxy resolution occurs only when you make a remote call, e.g. when you call checkedCast.

Best regards,
Bernard

aswani

Hi Bernard,

Thanks for clarifying that StringToProxy is a purely local operation.

The problem we are having is that it works for sometime and then does not respond (irrespective of the load). I have attached the configuration files we are using for icegrid registry, node.
At the client we use
icegridlocator=IceGrid/Locator:tcp -h 172.22.32.222 -p 4061
and we have all network traces on.

We are running RedHat Server on VmWare.
Are we using the right configurations?

Thanks,
Aswani

bernard

Hi Aswani,

Your configuration looks ok, but you're probably getting lots of tracing output. I was just interested in the client trace (--Ice.Trace.Network=2) when it fails to connect to the IceGrid registry. Could you attach it as well?

Also:

Ice.MessageSizeMax=10000000

This is ok (this just disables a check) but not necessary since your own operation's payload is not sent to the IceGrid node or registry. You only need to increase the MessageSizeMax for your own clients and servers.

IceGrid.Registry.DynamicRegistration=1

If you describe your servers and nodes in XML (or using the IceGrid GUI), you don't need this.

Best regards,
Bernard

aswani

Hi, Bernard

I am attaching the network traces of client and registry and also gdb traces of registry and node.

Cheers,
Aswani

benoit

Hi,

The stack traces of the IceGrid registry look wrong but this is most likely more a problem with gdb than IceGrid. The network traces don't show anything unusual, just that the client established successfully a connection to an Ice server.

Do you know if the IceGrid registry remains unavailable from your Web server machine for a long time once you get this connection refused exception? Or does it start working again shortly after for another request?

Could it be possible that the network interface sometime goes down and up again on your VMware machine (you should see some tracing in /var/log/messages if that's the case)?

One other thing I noticed, in your application descriptor you have the following:

<adapter name="outduICEUserAdapter" id="outduICEUserAdapter" endpoints="tcp"/>

I would recommend to also set the "-h <host>" endpoint option for the tcp endpoint here to ensure that only the given IP address gets published by the server.

Cheers,
Benoit.

aswani

Hi Benoit,

The IceGrid Registry does seem to remain unavailable from the Web Server for a long time after we get the connection refused exception. Sometimes, it does start working again after a few minutes, but at other times it never works.

The VMWare network seems be to alright, as there are no logs in /var/log/messages.

I will add the -h endpoint in the application descriptor. Will this solve cases where the node was unreachable for sometime?

Thanks,
Aswani