Archived

This forum has been archived. Please start a new discussion on GitHub.

IceGrid 3.3.0 unresponsive

I'm having some problems running IceGrid based apps across two computing nodes.

I initially developed multi-application system on a single (Node A) platform (Windows XP), using IceGrid to launch the apps. Node A hosts IceGridNode and IceGridRegistry. All was good.

I then re-configured IceGrid to run one of the apps on a seperate Windows XP platform (Node B). However, I have been largely unsuccessful in this endeavour. I then tried running Node B as a Linux (SUSE 10.1), but with even less success.

With Node B as Windows I observed the following:
1) Using the IceGridGUI app (on Node A): Node B could take a minute or two to connect.
2) Using IceGridAdmin to update the configuration resulted in an update to the Ice config file on the remote node db/node/servers/ServerB.
3) When ServerB was launched by IceGrid on Node B it was very, very sluggish and did not appear to run reliably at all.

With Node B as Linux I observed the following:
1) Using the IceGridGUI app (on Node A): Node B could take a minutes to connect. When connected, IceGridGUI was not able to obtain the resource utilisation data from Node B.
2) Using IceGridAdmin to update the configuration resulted in NO update to the Ice config file on the remote node db/node/servers/ServerB.
3) Could not start ServerB on Node B even using IceGridAdminGUI.
4) I then set Ice.Trace.Network=2 and noticed icgridnode was able to connect to the IceGridregistry node (192.168.0.101) as well as try and connect to other endpoints located on my VMWare internal network (192.168.15.1:1755 (VMnet8), 192.168.220.1:1755 (VMnet1)) - Is IceGridNode doing this?? it explains why Icegridnode on Node B is unresponsive - I assume blocked waiting for a socket connection to non-existent endpoints on VMWare network?? I have VMWare Player 2.0.4 installed on NodeA.

Below is my configuration details:
NodeA, IP address = 192.168.0.101
NodeB, IP address = 192.168.0.102

Node A config.grid file contents are:
IceGrid.InstanceName=DemoIceGrid
Ice.Default.Locator=DemoIceGrid/Locator:tcp -h NodeA -p 4061
IceGrid.Registry.DefaultTemplates=C:\Ice-3.3.0-VC90\config\templates.xml
IceGrid.Registry.Client.Endpoints=tcp -p 4061
IceGrid.Registry.Server.Endpoints=tcp
IceGrid.Registry.Internal.Endpoints=tcp
IceGrid.Registry.Data=..\db\registry
IceFrid.Registry.DynamicRegistration=1
IceGrid.Registry.PermissionsVerifier=DemoIceGrid/NullPermissionsVerifier
IceGrid.Registry.AdminPermissionsVerifier=DemoIceGrid/NullPermissionsVerifier
IceGrid.Registry.SSLPermissionsVerifier=DemoIceGrid/NullSSLPermissionsVerifier
IceGrid.Registry.AdminSSLPermissionsVerifier=DemoIceGrid/NullSSLPermissionsVerifier
IceGrid.Node.Name=Local
IceGrid.Node.Endpoints=default
IceGrid.Node.Data=..\db\node
IceGrid.Node.CollocateRegistry=1
IceGrid.Node.Trace.Activator=1
IceGridAdmin.Username=foo
IceGridAdmin.Password=bar

Node B (Linux) config.grid file contents are:
IceGrid.InstanceName=DemoIceGrid
Ice.Default.Locator=DemoIceGrid/Locator:tcp -h NodeA-p 4061
IceGrid.Node.Name=Remote
IceGrid.Node.Endpoints=default
IceGrid.Node.Data=/home/johnha/progam/db/node
IceGrid.Node.CollocateRegistry=0
IceGrid.Node.Trace.Activator=1
IceGrid.Node.Trace.Adapter=2
IceGrid.Node.Trace.Server=3
Ice.Trace.Network=2

Comments

  • benoit
    benoit Rennes, France
    Hi John,

    As you discovered, the registry or nodes are sometime failing to connect to IP addresses which are unreachable (the VMware IP addresses) and this explains the delays.

    By default, if you don't specify any "-h <host>" option in an objet adapter Endpoints property, the object adapter will listen on all local interfaces and if you don't set the the object adapter PublishedEndpoints property, the proxies published by the object adapter will contain the IP addresses of all the local interfaces. This is what is happening here, the registry or nodes are publishing proxies containing the VMware IP addresses.

    Adding -h 192.168.0.101 or -h 192.168.0.102 to each <name>.Endpoints property in your registry and node configuration files will solve the problem (the registry/nodes will publish proxies containing only the given IP address). I also recommend setting timeouts. For example:
    // Node A
    Ice.Default.Locator=DemoIceGrid/Locator:tcp -h NodeA -p 4061 -t 10000
    IceGrid.Registry.Client.Endpoints=tcp -p 4061 -h 192.168.0.101 -t 10000
    IceGrid.Registry.Server.Endpoints=tcp -h 192.168.0.101 -t 10000
    IceGrid.Registry.Internal.Endpoints=tcp -h 192.168.0.101 -t 10000
    IceGrid.Node.Endpoints=tcp -h 192.168.0.101 -t 10000
    
    // Node B
    Ice.Default.Locator=DemoIceGrid/Locator:tcp -h NodeA -p 4061 -t 10000
    IceGrid.Node.Endpoints=tcp -h 192.168.0.102 -t 10000
    

    Cheers,
    Benoit.
  • 1 step forward, 2 back

    Adding the host qualification has resolved the IceGrid service connection reliability. However I have now unearted another problem. When my client application attempts to connect to each server I get the following exception:

    EM: Checking for FesControl-1
    EM: exception - Ice.ConnectionRefusedException
    error = 0
    at IceInternal.ProxyFactory.checkRetryAfterException(LocalException ex, Reference ref, OutgoingAsync outAsync, Int32& cnt) in d:\builds\distbuilds\release\Ice-3.3.0\cs\src\Ice\ProxyFactory.cs:line 196
    at Ice.ObjectPrxHelperBase.handleException__(ObjectDel_ delegate, LocalException ex, OutgoingAsync outAsync, Int32& cnt) in d:\builds\distbuilds\release\Ice-3.3.0\cs\src\Ice\Proxy.cs:line 880
    at Ice.ObjectPrxHelperBase.ice_isA(String id__, Dictionary`2 context__, Boolean explicitContext__) in d:\builds\distbuilds\release\Ice-3.3.0\cs\src\Ice\Proxy.cs:line 168
    at Ice.ObjectPrxHelperBase.ice_isA(String id__) in d:\builds\distbuilds\release\Ice-3.3.0\cs\src\Ice\Proxy.cs:line 139
    at SwSim.RunControlPrxHelper.checkedCast(ObjectPrx b) in C:\swsim\DDS\generated\sim.cs:line 1484
    at SimController.Add(String serverId) in C:\swsim\ExecManager\SimController.cs:line 25
    Caused by: System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:4061
    at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
    at IceInternal.Network.doEndConnectAsync(IAsyncResult result) in d:\builds\distbuilds\release\Ice-3.3.0\cs\src\Ice\Network.cs:line 684

    My grid.xml snippet is:
    [HTML]<server-template id="FesServer">
    <parameter name="index"/>
    <server id="FesServer-${index}" exe="\swsim\bin\fes.exe" activation="on-demand">
    <adapter name="EsmSim" endpoints="tcp:udp" register-process="true">
    <object identity="FesControl-${index}" type="::SwSim::RunControl" property="ControlIdentity"/>
    <object identity="FesScenario-${index}" type="::SwSim::Environment" property="ScenarioIdentity"/>
    </adapter>
    </server>
    </server-template>[/HTML]

    The code that sets up the proxy to the server is:
    public void Add(string serverId)
        {
            RunControlPrx controller;
    
            try
            {
                Trace.WriteLine("EM: Checking for " + serverId);
                controller = RunControlPrxHelper.checkedCast(IceHelper.Communicator().stringToProxy(serverId));
            }
            catch (Ice.NotRegisteredException)
            {
                Trace.WriteLine("EM: " + serverId + "not registered");
                IceGrid.QueryPrx query =
                    IceGrid.QueryPrxHelper.checkedCast(IceHelper.Communicator().stringToProxy("DemoIceGrid/Query"));
    
                Ice.Identity id = IceHelper.Communicator().stringToIdentity(serverId);
                controller = RunControlPrxHelper.checkedCast(query.findObjectById(id));
            }
            catch (Ice.Exception ex)
            {
                Trace.WriteLine("EM: exception - " + ex.ToString());
                return;
            }
    
            if (controller == null)
            {
                Trace.WriteLine("EM: couldn't find " + serverId);
                return;
            }
            Trace.WriteLine("EM: Located: " + serverId);
    
            m_controllers.Add(controller);
        }
    
  • xdm
    xdm La Coruña, Spain
    Hi John

    The easy way to debug your problem is set Ice.Trace.Network=3 in your client configuration, this will print debug info that permit you view what is happening, most probable the client is trying to connect to a incorrect ip address or a firewall is blocking the connection.

    Regards,
    José
  • benoit
    benoit Rennes, France
    Hi,

    The exception shows that your client can't connect to the IceGrid registry with the IP address 127.0.0.1 and port 4061. Is the registry running on the same machine? How do you configure the Ice.Default.Locator property for your client? You most likely need to specify the IP address of the machine where the IceGrid registry is running in the locator proxy endpoints, for example:
    Ice.Default.Locator=DemoIceGrid/Locator:tcp -h 192.168.0.101 -p 4061 -t 10000
    

    Cheers,
    Benoit.
  • oops - found the error of my ways

    I overlooked modification to initialisation code in my client. I changed it from:
    // Initialise Ice
    Ice.Properties props = Ice.Util.createProperties();
    props.setProperty("IceStorm.TopicManager.Proxy", "IceStorm/TopicManager");
    props.setProperty("Ice.Default.Locator", "DemoIceGrid/Locator:default -p 4061");
    Ice.InitializationData id = new Ice.InitializationData();
    id.properties = props;
    communicator = Ice.Util.initialize(id);
    

    to:
    // Initialise Ice
    Ice.Properties props = Ice.Util.createProperties();
    props.setProperty("IceStorm.TopicManager.Proxy", "IceStorm/TopicManager");
    props.setProperty("Ice.Default.Locator", "DemoIceGrid/Locator:tcp -h 192.168.0.101 -p 4061");
    Ice.InitializationData id = new Ice.InitializationData();
    id.properties = props;
    communicator = Ice.Util.initialize(id);
    

    and now things are working fine for the Windows environment.

    I am still unable to start a server running on the Linux platform (NodeB). When I run icegridadmin to update the registry it appears to be sending an update of the config file to NodeB (by looking at trace information). However, the db/node/servers folder remains empty. Perhaps it is a permissions thing. Do I need to run IceGridNode as root on the Linux box? Is there a trace setting to examine updates to the db folders.
  • bernard
    bernard Jupiter, FL
    Hi John,

    The IceGrid node needs to have write permissions on its data directory (/home/johnha/progam/db/node for your NodeB). It does not need to run as root.

    When you deploy a server on NodeB (with NodeB running and registered with your IceGrid registry), you should see the server's configuration file appear in ...node/servers.

    You could run the IceGrid GUI Admin and connect to your IceGrid registry to see what's going on. Make sure the registry sees NodeB as running, and perhaps undeploy and then redeploy this server.

    Best regards,
    Bernard
  • Damn that firewall

    Having the firewall enabled (with my original settings) on NodeB (Linux) was blocking connection attempts from IceGrid services running on NodeA.

    After disabling the firewall all appears good....hooray!