Archived

This forum has been archived. Please start a new discussion on GitHub.

IceGrid Questions

Hello ZeroC,

I have a few questions as I try to get my application working with IceGrid.

Firstly, is it possible for a Glacier2 client to obtain a list of the SessionManagers registered with the locator? I would like clients to have the option to choose to which server they are connecting. I have to use a SessionManager per server since servers are spread around the internet in different locations.

To help me with my next question, you will probably need to see my configuration, so here is the important part:
<node name="node1">
         <server id="Bags2" activation="on-demand" exe="C:\Documents and Settings\pete\Desktop\Pelican Repo\lbags\Debug\lbags.exe">
            <properties>
            </properties>
            <adapter name="Bags" endpoints="tcp" id="${server}.Bags" register-process="true"/>
         </server>
         <server-instance template="Glacier2" client-endpoints="tcp -p 4502 -t 10000" server-endpoints="tcp -h 127.0.0.1" session-timeout="60">
            <properties>
               <property name="Glacier2.PermissionsVerifier" value="Glacier2/NullPermissionsVerifier"/>
            </properties>
         </server-instance>
         <server-instance template="IceStorm"/>
      </node>

When using this config why is it that the command "object list" only returns these objects?
bags.IceStorm/TopicManager
IceGrid/Query
IceGrid/Locator
IceGrid/Registry
IceGrid/InternalRegistry-Master

Shouldn't the Glacier and Bags2 be registered with the locator? In my IceGrid Admin GUI, they're both running.

Thanks a bunch,
Pete

Comments

  • benoit
    benoit Rennes, France
    Hi Pete,

    There are two possibilities to deploy the IceGrid registry when using Glacier2:
    • the IceGrid registry client endpoints are only accessible from the internal network behind the firewall and/or Glacier2. The locator proxy is configured with Ice.Default.Locator in the Glacier2 configuration file (clients don't need to have Ice.Default.Locator set in their configuration, Glacier2 takes care of resolving indirect proxy endpoints on behalf of the client). Advantage: you only need to expose the Glacier2 client endpoints on the public network, IceGrid is inaccessible from the outside. Disadvantage: clients need to establish a session to be able to use the locator or query the IceGrid registry.
    • the IceGrid client endpoints are accessible to clients on the public network. In this case, the locator proxy can be configured with Ice.Default.Locator in the client configuration. Advantage: the client can resolve indirect proxy endpoints using the locator or query the IceGrid registry without having a Glacier2 session established. Disadvantage: the IceGrid registry client endpoints need to be accessible from the public network and therefore any client can use the locator or query the IceGrid registry.

    So to answer your question, yes it's possible for the client to retrieve the list of session managers but depending on which kind of deployment you choose for the registry, the client might need to first establish a session to a Glacier2 instance to be able to use the locator or query the IceGrid registry.

    As for your second question, only few objects are registered with the IceGrid registry: objects explicitly declared as well-known objects in the IceGrid descriptors. See the following link in the Ice manual for more information: http://www.zeroc.com/doc/Ice-3.3.1/manual/IceGrid.40.6.html

    Cheers,
    Benoit.
  • Thanks Benoit, I'm slowly starting to get my head wrapped around how IceGrid can be used with my application.

    I think that I will have to leave the registry endpoints open because clients must determine the least loaded node, and a list or nodes before establishing a session.

    I'm having a problem that I'm wondering if someone can point me in the right direction. My Glacier cannot see the session manager. I'm getting this error message:
    [ 03/25/09 20:19:00.781 bags.Glacier2: Network: attempting to bind to tcp socket 127.0.0.1:0 ]
    [ 03/25/09 20:19:00.796 bags.Glacier2: Network: accepting tcp connections at 127.0.0.1:1168 ]
    [ 03/25/09 20:19:01.406 bags.Glacier2: Network: trying to establish tcp connection to 74.63.67.68:12000 ]
    [ 03/25/09 20:19:01.421 bags.Glacier2: Network: tcp connection established
      local address = 74.63.67.68:1171
      remote address = 74.63.67.68:12000 ]
    [ 03/25/09 20:19:01.421 bags.Glacier2: Network: trying to establish tcp connection to 74.63.67.68:1042 ]
    [ 03/25/09 20:19:01.421 bags.Glacier2: Network: tcp connection established
      local address = 74.63.67.68:1176
      remote address = 74.63.67.68:1042 ]
    [ 03/25/09 20:19:01.421 bags.Glacier2: Locator: registered server `bags.Glacier2' with the locator registry ]
    [ 03/25/09 20:19:01.421 bags.Glacier2: Network: attempting to bind to tcp socket 0.0.0.0:4502 ]
    [ 03/25/09 20:19:01.421 bags.Glacier2: Network: accepted tcp connection
      local address = 127.0.0.1:1168
      remote address = 127.0.0.1:1177 ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Network: accepting tcp connections at 0.0.0.0:4502 ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Network: attempting to bind to tcp socket 127.0.0.1:0 ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Network: accepting tcp connections at 127.0.0.1:1178 ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Network: created adapter `Glacier2Internal.Verifiers' without endpoints ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Locator: searching for object by id
      object = Glacier2/NullPermissionsVerifier ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Locator: object not found
      object = Glacier2/NullPermissionsVerifier ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Locator: searching for object by id
      object = Glacier2/NullPermissionsVerifier ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Locator: object not found
      object = Glacier2/NullPermissionsVerifier ]
    03/25/09 20:19:01.437 bags.Glacier2: warning: unable to contact permissions verifier `Glacier2/NullPermissionsVerifier'
    LocatorInfo.cpp:759: Ice::NotRegisteredException:
     no object with id `Glacier2/NullPermissionsVerifier' is registered
    [ 03/25/09 20:19:01.437 bags.Glacier2: Locator: searching for object by id
      object = Bags2.Bags ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Locator: retrieved endpoints from locator, adding to locator table
      object = Bags2.Bags
      endpoints = tcp -h 74.63.67.68 -p 1165 ]
    [ 03/25/09 20:19:01.437 bags.Glacier2: Network: trying to establish tcp connection to 74.63.67.68:1165 ]
    [ 03/25/09 20:19:01.453 bags.Glacier2: Network: tcp connection established
      local address = 74.63.67.68:1179
      remote address = 74.63.67.68:1165 ]
    03/25/09 20:19:01.453 bags.Glacier2: warning: unable to contact session manager `Bags2.Bags'
    Outgoing.cpp:422: Ice::ObjectNotExistException:
    object does not exist:
    identity: `Bags2.Bags'
    facet: 
    operation: ice_isA
    

    So it appears that the connection is established with Bags2.Bags, but ice_isA fails. I think ice_isA is trying to determine if it is a Glacier2::SessionManager? I'm guessing that the Glacier doesn't think that this object contains a session manager. Here is my application config:
         </server-template>
          <node name="node1">
             <server id="Bags2" activation="on-demand" exe="C:\Documents and Settings\pete\Desktop\Pelican Repo\lbags\Debug\lbags.exe">
                <properties>
                </properties>
               <adapter name="Bags" endpoints="tcp" id="${server}.Bags" register-process="true">
               <object identity="${server}.Bags" type="::Glacier2::SessionManager"/>
               </adapter>
             </server>
             <server-instance template="Glacier2" client-endpoints="tcp -p 4502 -t 10000" server-endpoints="tcp -h 127.0.0.1" session-timeout="60">
                <properties>
                   <property name="Glacier2.PermissionsVerifier" value="Glacier2/NullPermissionsVerifier"/>
                </properties>
             </server-instance>
             <server-instance template="IceStorm"/>
          </node>
    

    And inside the Bags2 object, I do implement a session manager as shown here:
    Glacier2::SessionPrx
    SessionManagerI::create(const string& name,
                                const Glacier2::SessionControlPrx& sessionControl,
                                const Ice::Current& c)
    {
        Glacier2::SessionPrx proxy;
        try
        {
            SessionIPtr session = new SessionI(name);
            proxy = Glacier2::SessionPrx::uncheckedCast(c.adapter->addWithUUID(session));
        }
        catch(const Ice::LocalException&)
        {
            if(proxy)
                proxy->destroy();
            
            throw Glacier2::CannotCreateSessionException("internal server error");
        }
        return proxy;
    }
    

    I feel that I must be missing an obvious thing.

    Thanks,
    Pete
  • benoit
    benoit Rennes, France
    Hi,

    About this error:
    03/25/09 20:19:01.437 bags.Glacier2: warning: unable to contact permissions verifier `Glacier2/NullPermissionsVerifier'
    LocatorInfo.cpp:759: Ice::NotRegisteredException:
     no object with id `Glacier2/NullPermissionsVerifier' is registered
    

    If your using the Glacier2 template from your Ice distribution, then you should most likely set Glacier2.PermissionsVerifier to bags.Glacier2/NullPermissionsVerifier since the instance name of the Glacier2 service is "bags.Glacier2" (according to the traces).

    Regarding this error:
    03/25/09 20:19:01.453 bags.Glacier2: warning: unable to contact session manager `Bags2.Bags'
    Outgoing.cpp:422: Ice::ObjectNotExistException:
    object does not exist:
    identity: `Bags2.Bags'
    facet: 
    operation: ice_isA
    

    It indicates that Glacier2 tried to check that the configured session manager proxy implements the Glacier2::SessionManager interface but it failed because the Ice object pointed by the configured proxy doesn't exist in the server.

    According to well known object declared in your server descriptor, the session manager object identity is Bags2.Bags (${server}.Bags). Your server should use this identity to add the servant to the Ice object adapter, most likely it's adding the servant with another identity right now and that's why the Glacier2 service gets the ObjectNotExistException.

    Note that you can easily generate a property for the well-known object with the property attribute, for example:
      <object identity="${server}.Bags" type="::Glacier2::SessionManager" property="SessionManagerIdentity"/>
    

    Then, it's easy to use this property to retrieve the session manager identity and register the servant with the object adapter:
       Ice::CommunicatorPtr com = ...;
       Ice::ObjectAdapterPtr adapter = ...;
       string idProp = com->getProperties()->getProperty("SessionManagerIdentity");
       adapter->add(new SessionManagerI(), com->stringToIdentity(idProp));
    

    Btw, if you're only using Ice 3.3.x, you can remove the register-process="true" attribute in the adapter XML element, this attribute is only useful for Ice servers using Ice < 3.3.

    Cheers,
    Benoit.
  • Thank you Benoit! You are a great help to me!

    I have resolved all of those issues by following your instructions. Now I believe that I need to make my Glacier a well-known object, so the client can resolve its proxy from the locator and obtain a connection.

    I looked through the server-template for Glacier2 that ships with Ice, and I see that it doesn't have the adapter tag where I would normally add an object child to make it well-known. If the client tries to resolve bags.Glacier2, I just get the NotRegisteredException. I expected this since bags.Glacier2 didn't show up when I ran "object list".

    How should I go about making my Glacier well-known?

    Thanks!
    Pete
  • benoit
    benoit Rennes, France
    Hi Pete,

    I've been discussing your deployment with a colleague and we feel that it's not a good idea to expose the IceGrid registry on the public network. There's a number of reasons for this:
    • it's more complicated: your client will have to make un-routed calls initially to find the Glacier2 router and then routed calls once it establishes the session.
    • you need to expose the IceGrid registry on the public network and it will therefore be subject to DoS attacks and this could end-up affecting your back-end servers.
    • it doesn't really bring you much. Using random load balancing would be simpler and probably as good as trying to find the least loaded Glacier2 instance.

    So my recommendation would be to start with something simple and not expose the IceGrid registry on the public network. Instead, your client could use a DNS name such as "glacier2.foo.com" to connect to Glacier2. This DNS name would be configured (with the DNS server) to resolve to one of the IP addresses of your deployed Glacier2 routers (the DNS lookup will in effect take care of the random load balancing). This way if you add/remove Glacier2 instances, you only have to update the DNS server configuration. Of course, you can still use IceGrid to deploy and manage the Glacier2 server and session manager server at a given site.

    Also, you should make sure to setup appropriate filtering for Glacier2 to make sure clients can only access Ice objects they are supposed to use.

    Cheers,
    Benoit.
  • Hi Benoit,

    Thanks for the advice specific to our environment! I've been thinking about your advice. To meet our deadline, you should know that we have decided against trying to make a grid that looks like one big server and instead provide users with a list of servers that they can choose from. Therefore, client partitioning does exist, at least for our initial release.

    We will start with just one server and expand as needed. We are planning to offer users a recommended server (least loaded), but they can still choose a specific server (for instance if his/her friend is playing on it). In addition, we would place a max user cap on each server, but I'm not sure what that number is yet. I think Matthew did something about this in his blog, I'll have to read it soon.

    I think random load balancing with DNS would work well with our old plan, but it seems that the DNS solution would limit us from allowing users to select their server. This would make it so two friends could only play together by luck. What we really need is a way to get a list of all the Glacier2 proxies available, so I'm not really sure what to do if a IceGrid registry is a bad option. What if we dedicated a server to just serve the registry behind Glacier2? Or simpler, just a PHP script that returns a list of Glacier2 proxies and its respective load?

    Thanks again!
    Pete
  • benoit
    benoit Rennes, France
    Hi Pete,

    Yes, a service dedicated to providing the Glacier2 proxies is probably better then if you need to provide the option to choose between different Glacier2 routers. It would be a very simple service, something like the following perhaps:
    // Slice
    dictionary<string, Glacier2::Router*> RouterPrxDist;
    interface Glacier2Lookup
    {
         Glacier2::Router* get();
         RouterPrxDict getAll();
    };
    

    The implementation of get() could return a random router or the least loaded one if you have this information. You could simply configure the list of Glacier2 routers in the configuration file of this service. And to avoid a single point of failure, the best would be to deploy this service on all the different sites and have clients use a DNS name to resolve it (like I suggested above for Glacier2).

    A PHP script that returns the list of Glacier2 routers would work too but it's probably just as simple to implement an Ice server for this (no need to parse the output of the PHP script in the client :)).

    Something which isn't clear to me, how are you going to deploy IceGrid at the different sites? Do you plan to deploy independent IceGrid "domains" at each site or will you share a same IceGrid domain between the different sites? By IceGrid domain, I'm referring to the set of IceGrid nodes and registries responsible to manage your application.

    If the different sites share a same IceGrid domain, the IceGrid nodes and registries will need to be able to communicate between each other. This means you'll need to open ports on your firewall at each site for the different IceGrid endpoints. These endpoints will need to be secured with IceSSL to make sure only authorized components can access them (you don't want a malicious client to stop your back-end servers!).

    It might be simpler to deploy different IceGrid domains at each site (i.e.: you deploy an IceGrid registry master and node at each site and there's no communications between the nodes/registries from different site). This way, no need to open firewall ports for IceGrid -- the components at a given site are totally independent from the ones at other sites. The disadvantage of this latest solution is that you can't have an overall view of your deployment at the different sites (with the IceGridGUI), you'll need to connect at each site...

    Cheers,
    Benoit.
  • Hi Benoit,

    This sounds like a wonderful solution! I particularly like the idea of running this lookup service on all of the nodes with random load balancing via DNS. This leaves no single point of failure! This service can also exist behind Glacier2 to avoid un-routed calls. This is definitely what I'm going to do.

    As far as IceGrid goes, I think that I'll run completely independent nodes. We will only have one node to start with. I imagine that one server can handle thousands of clients at a time, so the other option doesn't buy us anything unless our game does really well. I really am clueless about how many servers we will end up needing. If it becomes quite a few, I suppose we can always switch later rather easily. Anyhow, that's a problem that I'd love to have :) .

    Thank you, Benoit, for providing me some really great advice. The best solution is now clear to me.

    Pete
  • benoit
    benoit Rennes, France
    slypete wrote: »
    Hi Benoit,
    This service can also exist behind Glacier2 to avoid un-routed calls. This is definitely what I'm going to do.

    If it's only accessible through Glacier2, you'll need to first establish a session with a random Glacier2 router to get the list of the routers and then eventually close this session to a open a new one using a different router -- exposing this service on the public network would be simpler. Your client just need to do something like the following:
    // C++
    
    Glacier2LookupPrx lookup = Glacier2LookupPrx::unchecedCast(communicator->stringToProxy("lookup:tcp -p 12345 -h lookup.foo.net"));
    Glacier2::RouterPrx router = lookup->get();
    
    communicator->setDefaultRouter(router);
    
    ... establish the session with router ...
    

    Cheers,
    Benoit.
  • Is there a problem with making un-routed calls from the iPhone? I know you mentioned this as a drawback previously. Was it something specific to IceGrid?

    Is it possible that this public service may not be accessible to a client behind a strict firewall? I believe that a standard firewall configuration allows a response inbound if the client first requested something outbound. So, this should be OK, am I correct?

    Thanks,
    Pete
  • benoit
    benoit Rennes, France
    It's not really an issue, just a bit more complicated. You can make un-routed and routed calls in the client as long as you're careful and understand which invocations will be routed and which ones won't be :)

    Basically, an invocation will be routed if the proxy has an Ice router set. By default, when a proxy is created, it's using the default router set on the communicator. So, once you called setDefaultRouter on the communicator all proxies created after the call will use this router and proxies created before won't...

    Cheers,
    Benoit.
  • benoit
    benoit Rennes, France
    Sorry, forgot to answer your second question. Yes, clients behind a firewall can usually establish outbound connections but not accept incoming connections. The lookup service doesn't call on the client so that's fine (i.e.: it doesn't need to establish a connection to the client).

    Benoit.
  • Ok, Benoit. Thanks for clearing that up in my brain.

    Just in case you're interested, I'm thinking a little further about my IceGrid setup. I'll probably have to setup the IceGrid nodes to work together. I think this is worth while since all of the lookup service instances would be able to contact (or the other way around, not sure yet), obtain, and cache load stats of all the nodes. Perhaps the Glacier2 lookup could even use the locator in its implementation!

    Thanks a lot! :D
    Pete
  • benoit
    benoit Rennes, France
    Yes, that's one advantage of having a single IceGrid domain across the different sites but then the setup is more complicated. Depending on your time constraints, you could go for the simple solution first and improve it later... (the good thing is that you can change afterwards without changes for existing clients).

    Cheers,
    Benoit.