Archived

This forum has been archived. Please start a new discussion on GitHub.

Application add is very slow, what happened?

eaglecn
eaglecn
in Help Center
Hi,
We use icegrid to manage several applications, find that adding some an application is slow. So what happened when adding an application or how can we trace these activities?
thanks!

Comments

  • benoit
    benoit Rennes, France
    Hi,

    You can add the following traces on the registry:

    IceGrid.Registry.Trace.Server=3
    IceGrid.Registry.Trace.Object=1
    IceGrid.Registry.Trace.Adapter=1

    This should trace the newly added objects, adapters and servers and the deployment of each server on the nodes.

    Cheers,
    Benoit.
  • eaglecn
    eaglecn
    I add cout code to function of "IceProxy::IceGrid::Admin::addApplication" in file of Ice-3.2.1/src/IceGrid/Admin.cpp just now as following, 
                cout<<"before checktwoway=="<<endl;
            __checkTwowayOnly(__IceGrid__Admin__addApplication_name);
            cout<<"before getDelegate=="<<endl;
            __delBase = __getDelegate();
            cout<<"before dynamic_cast=="<<endl;
            ::IceDelegate::IceGrid::Admin* __del = dynamic_cast< ::IceDelegate::IceGrid::Admin*>(__delBase.get());
            cout<<"before _del->addApp=="<<endl;
            __del->addApplication(descriptor, __ctx);
    and find no exception.

    After tried several times, the same symptom reappears, namely, the add application command line is hanging.

    this time, it reports
    "IceGrid.Registry.Client.ThreadPool' is running low on threads" frequently,

    I got no idea with this parameter.
    after checking, the registry process has more than 18 threads.

    where is wrong?
  • benoit
    benoit Rennes, France
    Sorry, without more information it's impossible to say what could be wrong. Did you try adding the tracing suggested in my previous post? Can you copy/paste the tracing output here?

    Cheers,
    Benoit.
  • eaglecn
    eaglecn
    The following info is all I got from a rsh terminal, is it enough?
    the last application need 4 minutes to be loaded, while others almost need no time.

    [ 01/13/09 17:03:58.078 Adapter: added adapter `ddl_yf-softnode3.DDLAdapter' ]
    [ 01/13/09 17:03:58.079 Object: added object `ddl_yf-softnode3' ]
    [ 01/13/09 17:03:58.079 Server: added server `ddl_yf-softnode3' (`2C521B8F-2B06-4146-A121-4A128D8B8903', `1') ]
    [ 01/13/09 17:03:58.079 Server: loading `ddl_yf-softnode3' on node `yf-softnode3' ]
    [ 01/13/09 17:03:58.081 Server: loaded `ddl_yf-softnode3' on node `yf-softnode3' ]
    before checktwoway==
    before getDelegate==
    before dynamic_cast==
    before _del->addApp==
    [ 01/13/09 17:03:58.117 Adapter: added adapter `load_YF-softnode4.LoadAdapter' ]
    [ 01/13/09 17:03:58.117 Object: added object `load_YF-softnode4' ]
    [ 01/13/09 17:03:58.117 Server: added server `load_YF-softnode4' (`95E14DCE-AF6B-47D0-9515-BB0F952E0903', `1') ]
    [ 01/13/09 17:03:58.117 Server: loading `load_YF-softnode4' on node `YF-softnode4' ]
    [oracle@YF-softnode4 script]$ [ 01/13/09 17:07:07.092 Server: loaded `load_YF-softnode4' on node `YF-softnode4' ]

    and only this application locates at the other server
  • benoit
    benoit Rennes, France
    From the traces, it appears that the server loading on YF-softnode4 hangs since you're not getting the "Server: loaded `load_YF-softnode4' on node `YF-softnode4'" message. Is the host where YF-softnode4 runs accessible from the registry host?

    Also, you should add timeouts to the IceGrid registry and node endpoints to avoid requests to block for too long if something goes wrong. See the configuration files from cpp/demo/IceGrid/replication for an example on how to set timeouts.

    Cheers,
    Benoit.
  • eaglecn
    eaglecn
    these machines can access each other and not all loading of this application are not successful. In fact, adding process is slow but can be done finally.

    So, I am wonder during the hanging, what operation is ongoing? Is it just try to accessing remote server?

    Why other applications never have this problem? Processing properites in program has relation with it? But I think no processes are to be started during that time.
  • eaglecn
    eaglecn
    I got the following message when removing this application, 
    [ 01/13/09 17:21:51.357 Server: couldn't unload `load_YF-softnode4' on node `YF-softnode4':
  Network.cpp:624: Ice::ConnectTimeoutException:
  timeout while establishing a connection ]

[ 01/13/09 17:53:43.501 Server: couldn't unload `load_YF-softnode4' on node `YF-softnode4':
  Network.cpp:664: Ice::ConnectionRefusedException:
  connection refused: Connection refused ]
    why it is timout to establish a connection?
  • benoit
    benoit Rennes, France
    For some reasons, the registry can't contact the node YF-softnode4. You could try adding network tracing on the IceGrid registry with Ice.Trace.Network=2 to figure out on which IP address the registry tries to contact the node.

    If you didn't specify any host with the -h HOST option in the IceGrid node endpoints, it's possible that the registry tries to connect to an unreachable IP address if the host where the node is running has multiple network interfaces.

    You can try to specify explicitly the IP address in the node endpoints, for example:
       IceGrid.Node.Endpoints=tcp -h 192.168.1.2

    See 28.4.6 Endpoints in the Ice manual for more information on object adapter endpoints.

    Cheers,
    Benoit.
  • eaglecn
    eaglecn
    I believe the problem is solved after i set the icegridnode configure file.
    thanks a lot!

Categories