Slow grid start - node seems to be unreachable
Wilko
Member Wilko KempaOrganization: sd&m AGProject: System Management and Monitoring Plattform ✭
in Help Center
Hi Support Team
we currently test ICE 3.2.0 with a configuration that uses different applications running on two ice grid nodes "appserver" and "mediaserver". Following questions came up during the tests
We run Ice 3.2.0, at log level 2 we got the following output
At appserver (hosts registry, no applications deployed)
At mediaserver (no applications deployed)
Our appserver has the following config:
The mediaserver uses this config.
Thanks in advance
Wilko Kempa
sd&m AG
Common Service Plattform
we currently test ICE 3.2.0 with a configuration that uses different applications running on two ice grid nodes "appserver" and "mediaserver". Following questions came up during the tests
After starting both icegridnodes it takes >3min until mediaserver print its up message, Even if we have no apps deployed. As we have a load <0.3 on the servers and both are connected via GBit ethernet there seems no reason to be so slow. Do you have any ideas on how to improve this behaviour?
During debugging behaviour 1 another point came up, ice does not create a log even if we set the IceGrid.Node.Output property to an existing writable folder and the icegrid is runs as root.
Finally we need to start an apache that is started as root but switch to apache user after initialization. If ice grid starts this process (running as root) it prints a message "server is not allowed to run as root". While this is a nice security feature - is there a way to overwrite this behaviour?
We run Ice 3.2.0, at log level 2 we got the following output
At appserver (hosts registry, no applications deployed)
[[email protected]_pia1 fridge]# ./run.sh Start IceGrid! [ 06/13/07 14:22:19.195 Node: node 'appserver' up ] [ 06/13/07 14:25:57.484 Node: node 'mediaserver' up ] ]
At mediaserver (no applications deployed)
[[email protected]_pia2 fridge]# ./run.sh Start IceGrid! [ 06/13/07 14:22:48.265 Replica: trying to establish session with replica Master' ] [ 06/13/07 14:25:57.390 Replica: established session with replica Master' ]
Our appserver has the following config:
# File config.icegrid # Registry configuration IceGrid.Registry.Client.Endpoints=tcp -p 12000 IceGrid.Registry.Server.Endpoints=tcp IceGrid.Registry.Internal.Endpoints=tcp IceGrid.Registry.Data=db/registry IceGrid.Registry.AdminPermissionsVerifier=IceGrid/NullPermissionsVerifier IceGrid.Registry.AdminCryptPasswords=/conf/authentications.txt IceGrid.Registry.Trace.Node=2 IceGrid.Registry.Trace.Locator=2 IceGrid.Node.RedirectErrToOut=1 IceGrid.Node.Output=/var/log/ice-applications/fridge # Only required if you want servers to register # themselves without explicit deployment. If all # servers are deployed explicitly, this property # can be left unset. IceGrid.Registry.DynamicRegistration=1 # Node configuration IceGrid.Node.CollocateRegistry=1 IceGrid.Node.Name=appserver IceGrid.Node.Endpoints=tcp IceGrid.Node.Data=db/node # Set the default locator so the node and admin # tools can find the registry. Ice.Default.Locator=IceGrid/Locator:tcp -h int1.ourdomain.de -p 12000
The mediaserver uses this config.
# File config.icegrid # Only required if you want servers to register # themselves without explicit deployment. If all # servers are deployed explicitly, this property # can be left unset. IceGrid.Registry.DynamicRegistration=1 # Node configuration IceGrid.Node.CollocateRegistry=0 IceGrid.Node.Name=mediaserver IceGrid.Node.Endpoints=tcp IceGrid.Node.Data=db/node IceGrid.Node.Trace.Replica=2 IceGrid.Registry.Trace.Locator=2 IceGrid.Node.RedirectErrToOut=1 IceGrid.Node.Output=/var/log/ice-applications/fridge # Set the default locator so the node and admin # tools can find the registry. Ice.Default.Locator=IceGrid/Locator:tcp -h int1.ourdomain.de -p 12000
Thanks in advance
Wilko Kempa
sd&m AG
Common Service Plattform
0
Comments
Here's the answers to your three questions:
IceGrid.Registry.Server.Endpoints=tcp
IceGrid.Registry.Internal.Endpoints=tcp
IceGrid.Registry.Server.Endpoints=tcp -h 192.168.1.2
IceGrid.Registry.Internal.Endpoints=tcp -h 192.168.1.2
Cheers,
Benoit.
For more information on securing IceGrid, you can to take a look at Matthew's newsletter article from the Connections issue #17.
Cheers,
Benoit.
Hi Benoit,
you will be pleased to hear your were right.
Best Regards
Wilko
Wilko Kempa
sd&m AG
Common Service Plattform
Icebox:
Grid:
Service:
I can't see anything wrong in the IceBox and IceGrid configuration files. In the service configuration file, you don't specify endpoints for the IceStorm.Publish.Endpoints property and the endpoints from Ice.Default.Locator don't match the ones you define for IceGrid (10.110.3.60 vs. 10.110.0.60).
I'm afraid without more information it's difficult to figure out what the problem could be. Could you detail when the timing issue occurs and for which servers? Did you run the servers with Ice.Trace.Network=2 to see if they aren't still trying to connect to some unreachable IP addresses?
Could you also detail your deployment? It sounds like you're using an IceGrid node with a collocated registry and an IceBox server with the IceStorm service but the IceBox server isn't managed by the IceGrid node. Is this correct?
Thanks,
Cheers,
Benoit.
The issue seems to be related to Linux networking alias. Eth0 is given two IP addresses and is divided into 2 devices, eth0 and eth0:0. Either, the eth0:0 device is giving the problems or the IP address given to eth0:0 is giving the problems. Do you know of any binding issues with aliases?
You are correct about the deployment. I will post the trace in a bit.
Cheers,
Benoit.
Here is the client output:
[ 3/5/08 18:41:29:173 Network: attempting to bind to tcp socket 10.110.3.10:0 ] [ 3/5/08 18:41:29:175 Network: accepting tcp connections at 10.110.3.10:52553 ] [ 3/5/08 18:41:29:176 Network: attempting to bind to tcp socket 127.0.0.1:0 ] [ 3/5/08 18:41:29:176 Network: accepting tcp connections at 127.0.0.1:40687 ] [ 3/5/08 18:41:29:182 Network: attempting to bind to udp socket 10.110.3.10:0 ] [ 3/5/08 18:41:29:183 Network: starting to receive udp packets local address = 10.110.3.10:32930 remote address = <not connected> ] [ 3/5/08 18:41:29:191 Network: attempting to bind to udp socket 127.0.0.1:0 ] [ 3/5/08 18:41:29:191 Network: starting to receive udp packets local address = 127.0.0.1:32932 remote address = <not connected> ] [ 3/5/08 18:41:29:194 Network: attempting to bind to ssl socket 10.110.3.10:0 ] [ 3/5/08 18:41:29:466 Network: accepting ssl connections at 10.110.3.10:42473 ] [ 3/5/08 18:41:29:466 Network: attempting to bind to ssl socket 127.0.0.1:0 ] [ 3/5/08 18:41:29:467 Network: accepting ssl connections at 127.0.0.1:35930 ] 18:41:29,474 [main] DEBUG com.brightsystemsinc.util.ClientAdapter - Object adapter set [ 3/5/08 18:41:30:908 Network: trying to establish tcp connection to 10.110.3.60:12000 ] [ 3/5/08 18:41:30:910 Network: tcp connection established local address = 10.110.3.10:48942 remote address = 10.110.3.60:12000 ] 18:41:39,924 [main] ERROR com.brightsystemsinc.bdrgui.BDRGUI - There was an error Ice.NoEndpointException proxy = "BDRDriveInfoService -t" at IceInternal.RoutableReference.createConnection(RoutableReference.java:418) at IceInternal.IndirectReference.getConnection(IndirectReference.java:181) at Ice._ObjectDelM.setup(_ObjectDelM.java:258) at Ice.ObjectPrxHelperBase.__getDelegate(ObjectPrxHelperBase.java:1050) at Ice.ObjectPrxHelperBase.ice_isA(ObjectPrxHelperBase.java:85) at Ice.ObjectPrxHelperBase.ice_isA(ObjectPrxHelperBase.java:61) at com.brightsystemsinc.bdrinterface.BDRInterface.BDRDriveInfo.BDRDriveInfoServicePrxHelper.checkedCast(BDRDriveInfoServicePrxHelper.java:297) at com.brightsystemsinc.display.CartridgeSetTabDisplay.init(CartridgeSetTabDisplay.java:134) at com.brightsystemsinc.display.CartridgeSetTabDisplay.<init>(CartridgeSetTabDisplay.java:106) at com.brightsystemsinc.bdrgui.BDRGUI.initFrame(BDRGUI.java:171) at com.brightsystemsinc.bdrgui.BDRGUI.run(BDRGUI.java:104) at Ice.Application.main(Application.java:114) at Ice.Application.main(Application.java:57) at com.brightsystemsinc.bdrgui.BDRGUI.main(BDRGUI.java:402)Server Output:
Icebox output:
Thanks for your help.
Could you please detail a little more what is your exact problem?
Both traces show that an attempt is made to connect to an unreachable address (1.53.1.8) and that the connection attempt hangs for about 2 minutes. This is expected if you don't use timeouts. You can set the Ice.Override.ConnectTimeout property if you want the connection attempt to timeout sooner.
However, the best is to avoid connection attempts to invalid addresses. You should find out which proxy contains this IP address and see if the server configuration can be changed so that it doesn't publish proxies with unreachable addresses.
Cheers,
Benoit.
Here are my configs:
Client:
Icebox:
Grid:
Icestorm:
As mentioned previously, you don't specify the host for the IceStorm.Publish.Endpoints property. Could this perhaps be the problem?
So if I understand it correctly, it's the client which is hanging when it tries to connect to IceStorm? Please correct me if I'm mistaken.
Did you figure out which Ice invocation was hanging by attaching with the debugger to the process while it hangs or by adding some tracing? Once you figure out where it hangs, you'll know which proxy is causing the hang and then you can check where the proxy is coming from and see if the configuration of the application that creates the proxy is correct.
Cheers,
Benoit.
The client is hanging on the following line:
This is a well-known proxy. How did you register this well-known proxy with the IceGrid registry? I suppose you used the icegridadmin "object add" command or the IceGrid GUI since IceStorm isn't managed by an IceGrid node, is this correct?
If this is the case, most likely, the -h option for the well-known proxy hasn't been specified when it was added to the IceGrid registry. You can check the endpoints of this well-known proxy with the "object describe" command or the IceGrid administrative GUI. Updating the endpoints of this well-known proxy to include the -h option should fix the hang.
Cheers,
Benoit.
Here is the line:
<server-instance template="IceStorm" instance-name="IceStorm"> </server-instance>I tried adding the endpoint property. It didn't help.
<server-instance template="IceStorm" instance-name="IceStorm"> <properties> <property name="IceStorm.Publish.Endpoints" value="default -p 20001:udp -p 20001 -h 10.110.3.60"/> </properties> </server-instance>I'm afraid I don't understand your deployment. I thought you were not deploying IceStorm on an IceGrid node with a deployment descriptor but instead was manually starting IceStorm from the command line using a configuration file. If you're indeed using a deployment descriptor to deploy IceStorm on an IceGrid node, you need to update the XML file for configuration changes (and run the icegridadmin "application update" command after each update). Could post your application.xml file? I suspect the problem is coming from the definition of the "endpoints" attribute for the TopicManager adapter (missing -h option).
Cheers,
Benoit.
<icegrid> <application name="BDR" import-default-templates="true"> <node name="localhost"> <server id="BDRServer" exe="/usr/bright/apps/bdr_server_start.sh" activation="always" user="root"> <adapter name="CpService" endpoints="tcp -h 10.110.3.60" register-process="true"> <object identity="BDRCpService" type="::BDRCp::BDRCpService" property="Identity"/> </adapter> <adapter name="CustomerInfoService" endpoints="tcp -h 10.110.3.60" register-process="true"> <object identity="BDRCustomerInfoService" type="::BDRCustomerInfo::BDRCustomerInfoService" property="Identity"/> </adapter> <adapter name="DriveInfoService" endpoints="tcp -h 10.110.3.60" register-process="true"> <object identity="BDRDriveInfoService" type="::BDRDriveInfo::BDRDriveInfoService" property="Identity"/> </adapter> <adapter name="CustomerInitService" endpoints="tcp -h 10.110.3.60" register-process="true"> <object identity="BDRCustomerInitService" type="::BDRCustomerInit::BDRCustomerInitService" property="Identity"/> </adapter> <adapter name="MountService" endpoints="tcp -h 10.110.3.60" register-process="true"> <object identity="BDRMountService" type="::BDRMount::BDRMountService" property="Identity"/> </adapter> <adapter name="UnmountService" endpoints="tcp -h 10.110.3.60" register-process="true"> <object identity="BDRUnmountService" type="::BDRUnmount::BDRUnmountService" property="Identity"/> </adapter> </server> <server-instance template="IceStorm" instance-name="IceStorm"> <properties> <property name="IceStorm.Publish.Endpoints" value="default -p 20001:udp -p 20001 -h 10.110.3.60"/> <property name="IceStorm.TopicManager.Endpoints" value="default -p 20000 -h 10.110.3.60"/> </properties> </server-instance> </node> </application> </icegrid>I have already ran the "appliation update" after updating my application.xml.
You can simply use the following instead:
Or:
<server-instance template="IceStorm" instance-name="IceStorm"> <properties service="IceStorm"> <property name="IceStorm.Publish.Endpoints" value="default -p 20001:udp -p 20001 -h 10.110.3.60"/> <property name="IceStorm.TopicManager.Endpoints" value="default -p 20000 -h 10.110.3.60"/> </properties> </server-instance>Make sure to run "application update" on the updated descriptor to push the changes to the IceGrid registry.
You can also use the "adapter endpoints IceStorm.TopicManager" command to check the runtime endpoints of the topic manager adapter once the IceBox server is started. In theory, it should only contain the 10.110.3.60 IP address.
Cheers,
Benoit.