indirect proxy and ObjectNotExistException

rc_hzrc_hz Member Eric RCOrganization: www.genband.comProject: No project yet ✭✭✭
I have noticed that when the Ice core in the client gets an ConnectionRefusedException, it will contact IceGrid again to refresh the locator cache and then use these new endpoints to make another try. However, when it gets an ObjectNotExistException, it will simply throw the exception. I think the latter will be misleading in some cases and sometimes will never recover. For example:
1.Suppose the OA1server's ObjectAdapter selects tcp port randomly:
OA1.Endpoints=tcp -h 10.10.10.1

Suppose OA1 listens on -p 10001 now.
OA1 also register dynamically to IceGrid.

2.Suppose the client is using indirect proxy([email protected]).  The client make an invocation and it gets an item from IceGrid:[email protected] --> [email protected] -h 10.10.10.1 -p 10001. Of course, the client's invocation succeeds.

3.Now the following events:
1)time1: The OA1's server restarts and listens on a new port: -p 10002
2)time2: Another server starts and it listens on -p 10001
3)time3: The previous client makes an invocation on [email protected] and it will get an ObjectNotException.

Comments

  • rc_hzrc_hz Member Eric RCOrganization: www.genband.comProject: No project yet ✭✭✭
    Another question about random ports

    Another question about random ports. Suppose we have the following configuration:
    1.IceGrid
    IceGrid.Registry.Client.Endpoints=tcp -p 10000
    

    2.Glacier2
    Glacier2.Client.Endpoints=tcp -p 20000
    

    3.non-Ice applications
    TcpListenPort=30000
    

    4.Normal Ice applications
    OA1.Endpoints=tcp -h 10.10.10.1
    OA2.Endpoints=tcp -h 10.10.10.1
    

    It means that OA1 and OA2 will select tcp port randomly. However, if we start OA1/OA2's server first, it will has the danger of using the specified ports: 10000/20000/30000.

    So it would be better if such an Ice property can be added:
    Ice.TcpPortExclude=10000,20000,20050-20099,30000
    
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    rc_hz wrote: »
    I have noticed that when the Ice core in the client gets an ConnectionRefusedException, it will contact IceGrid again to refresh the locator cache and then use these new endpoints to make another try. However, when it gets an ObjectNotExistException, it will simply throw the exception. I think the latter will be misleading in some cases and sometimes will never recover.

    The Ice::ObjectNotExistException is actually not thrown right away, the invocation is first retried by the Ice core (try again your scenario with enabling Ice.Trace.Retry). However, since the endpoints for the object adapter are still cached in the Ice core, the invocation will be tried again with the same server. You should disable the locator cache if you want this scenario to work.

    Invalidating the object adapter endpoints when an invocation fails with ObjectNotExistException could be another solution. However, this means that in the most common cases we might be invalidating the endpoints too often defeating the usefulness of the cache so I'm not sure it's a good idea.

    Cheers,
    Benoit.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    As for your second comment, most operating systems are using a specific range for "ephemeral ports" (ports allocated by the system, see here for more information) and sometime even allow you to configure this range. I think the best is to not use ports in the ephemeral port range to avoid any conflicts.

    Cheers,
    Benoit.
  • rc_hzrc_hz Member Eric RCOrganization: www.genband.comProject: No project yet ✭✭✭
    benoit wrote: »
    The Ice::ObjectNotExistException is actually not thrown right away, the invocation is first retried by the Ice core (try again your scenario with enabling Ice.Trace.Retry). However, since the endpoints for the object adapter are still cached in the Ice core, the invocation will be tried again with the same server. You should disable the locator cache if you want this scenario to work.

    Invalidating the object adapter endpoints when an invocation fails with ObjectNotExistException could be another solution. However, this means that in the most common cases we might be invalidating the endpoints too often defeating the usefulness of the cache so I'm not sure it's a good idea.

    Cheers,
    Benoit.

    Thank you. I mean: when the client's Ice core gets an ConnectionRefusedException, it will contact IceGrid again, so I think the client's Ice core can also contact IceGrid again if it gets an ObjectNotExistException.
    benoit wrote:
    As for your second comment, most operating systems are using a specific range for "ephemeral ports" (ports allocated by the system, see here (http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html) for more information) and sometime even allow you to configure this range. I think the best is to not use ports in the ephemeral port range to avoid any conflicts.
    This seems to work. Thanks.
  • benoitbenoit Rennes, FranceAdministrators, ZeroC Staff Benoit FoucherOrganization: ZeroC, Inc.Project: Ice ZeroC Staff
    rc_hz wrote: »
    Thank you. I mean: when the client's Ice core gets an ConnectionRefusedException, it will contact IceGrid again, so I think the client's Ice core can also contact IceGrid again if it gets an ObjectNotExistException.

    Yes, I understand what you mean. But note that IceGrid will contact the locator again if the invocation fails with an ObjectNotExistException AND if you disable the locator cache.

    The scenario you describe might happen when using the locator cache however. We could invalidate the adapter endpoints from the cache on ObjectNotExistException but it might have a performance impact on clients which are often expecting ObjectNotExistException from a server.

    Also, the scenario you describe is unlikely to happen if your servers are using ephemeral ports (assigned by the system). I believe most operating systems iterate over the ephemeral port range to assign ports for sockets, so to re-use a previously used ephemeral port you first have to use all the ports available in the ephemeral port range...

    Cheers,
    Benoit.
  • rc_hzrc_hz Member Eric RCOrganization: www.genband.comProject: No project yet ✭✭✭
    I understand now. Thank you very much.
Sign In or Register to comment.