Archived

This forum has been archived. Please start a new discussion on GitHub.

Question on scope of tags (as used in compact type ids and optional values)

Hi,

I just spent some time reading the release notes of 3.5 and stumbled over "compact type ids" and "optional data member and parameters".

It seems like that you're using a numeric tag to mark optional values. Besides the fact that this sounds dangerous in terms of collisions and feels hacky in general, my question is: What is the scope of these IDs? Are those only to be unique within the data structure they appear in, or the module, or system wide (shudder)?

Example from your wiki:
class Account
{
    string accountNo;
    ...
    optional(1) string guardian;
    optional(2) string linkedAccountNo;
};
 
interface Bank
{
    Account createAccount(string name, optional(3) string guardian, ...);
};

This seems to imply that all of these optional values are within the same scope of unique tags. Is this correct, or is it just a misleading example and doing the following would be valid and working:
class Account
{
    string accountNo;
    ...
    optional(1) string guardian;
    optional(2) string linkedAccountNo;
};
 
interface Bank
{
    Account createAccount(string name, optional(1) string guardian, ...);
};

An additional question is: Since those tags can also be used as short type IDs like in class:
class UniqueNumbersAre(1337)
{
  string collision(optional(1337) string boom);
};

Will this compile or will it break? In this case the scope would be the module - or are those numeric type ids supposed to be system wide (please tell me you didn't).

Some clarification would be nice.

Thanks,
Michael

Comments

  • benoit
    benoit Rennes, France
    Hi,

    Compact type IDs and the optional tags are un-related and therefore can't collide.

    Optional tags are scoped by the type they are defined in or the operation. See here for more information.

    The scope of Compact type IDs is the scope of the system or domain where the Ice applications are inter-connected. Compact type IDs are designed for those applications which want to minimize the size of the data on the wire. This is important for some applications deployed in environments where network bandwidth is costly. It is a lot less flexible than regular type IDs -- it should only used for specific cases. Typically, this can be used for the contract between an internet client and a front-end server (where the bandwidth might be costly). The back-end systems still use regular type IDs.

    Cheers,
    Benoit.
  • Hi Benoit,

    thanks for your quick and clear response.

    What does all of this mean for slice checksums?
    1. Are there slice checksums generated for compact type ids
    2. Will the slice checksum change for a struct or class when optional parameters/values are added? And if so, is there a (new) proposed way to to detect incompatible changes automatically? After all it would be interesting to be able to add optional values and maintain compatibility with an earlier version of the type and at the same time detect critical changes (like the addition/removal of a non-optional value/parameter).

    On a different note - but still related to 3.5 - I noticed your new (and more efficient) encoding format for classes/exceptions and have a few questions:
    • For migrating large projects from encoding v1.0, does it sound reasonable to annotate all base classes/exceptions to be "format:sliced" to avoid problems in the process and then remove those one by one after checking the code thoroughly?
    • I assume that the compact format has no impact on the way IceStorm is used (assuming it's a current IceStorm, like you're pointing out in the documentation)
    • I'm not completely clear what the new encoding really means for exceptions. Will it still be possible to catch a base exception when a more derived one has been thrown that's unknown to the caller? Will it be passed through if it's not caught by the caller? If not this might break existing code.

    To clarify my last question, imagine the following example (it doesn't really need slice level inheritance, different implementations of BaseI would be enough to get the same effect):

    Base.ice
    module Example
    {
      exception BaseException {};
    
      interface Base
      {
         void noop() throws BaseException;
      };
    
      interface BaseUser
      {
        void callNoop(Base* b) throws BaseException;
      }
    };
    

    Derived.ice:
    #include <Base.ice>
    
    module Example
    {
      exception DerivedException extends BaseException {};
    
      interface Derived extends Base
      {
        void anotherOp();
      }
    }; 
    

    BaseI.pseudocpp
    #include <Base.h>
    
    void BaseI::noop(const Ice::current&)
    {}
    

    BaseUserI.pseudcpp
    #include <Base.h>
    
    void BaseUserI::callNoop(const BasePrx& base, const Ice::Current&)
    {
      base->noop();
    }
    

    DerivedI.pseudocpp:
    #include <Derived.h>
    
    void DerivedI::noop(const Ice::Current&)
    {
      throw DerivedException();
    }
    

    Client.pseudocpp:
    #include <Derived.h>
    
    int main()
    {
      try
      {
        BaseUser bp = lookup(...);
        DerivedPrx dp = lookup(...);
        
        bp->callNoop(dp);
      }
      catch(DirectException&)
      {
        std::cout << "Direct" << std::endl;
      }
      catch(BaseException&)
      {
        std::cout << "Base" << std::endl;
      }
      catch(...)
      {
        std::cout << "Misc" << std::endl;    
      }
    }
    

    So, will the example above output "Direct", "Base" or "Misc"? Will the result depend on the encoding format (sliced vs compact)?

    Thanks,
    Michael
  • benoit
    benoit Rennes, France
    grembo wrote: »
    Hi Benoit,

    thanks for your quick and clear response.

    What does all of this mean for slice checksums?
    1. Are there slice checksums generated for compact type ids
    2. Will the slice checksum change for a struct or class when optional parameters/values are added? And if so, is there a (new) proposed way to to detect incompatible changes automatically? After all it would be interesting to be able to add optional values and maintain compatibility with an earlier version of the type and at the same time detect critical changes (like the addition/removal of a non-optional value/parameter).

    Note that Slice checksums are designed to provide a check for type equality not "on-the-wire" compatibility. So optionals parameters and data members are used in the checksum computation.

    However, the compact type ID isn't used for the checksum computation which is a bug. They should be taken into account. We will look into fixing this and provide a source patch. In the meantime, don't use compact type IDs if you use checksums or the fix will eventually break the check in the future.
    On a different note - but still related to 3.5 - I noticed your new (and more efficient) encoding format for classes/exceptions and have a few questions:
    • For migrating large projects from encoding v1.0, does it sound reasonable to annotate all base classes/exceptions to be "format:sliced" to avoid problems in the process and then remove those one by one after checking the code thoroughly?
    • I assume that the compact format has no impact on the way IceStorm is used (assuming it's a current IceStorm, like you're pointing out in the documentation)
    • I'm not completely clear what the new encoding really means for exceptions. Will it still be possible to catch a base exception when a more derived one has been thrown that's unknown to the caller? Will it be passed through if it's not caught by the caller? If not this might break existing code.

    Even for large projects, you know in general whether or not Ice client & servers rely on slicing. They generally potentially do if clients & servers use classes/exceptions and if they can use different versions of the Slice definitions (on-the-wire compatible definitions which in general means interfaces with additional operations, additional class/exception specializations).

    There are multiple solutions for the migration:
    • configure all the Ice 3.5.0 based applications with Ice.Default.SlicedFormat=1 and start using [format:compact] for the operations where it's safe and where you want to minimize the size of the Slice class instances.
    • do as you suggest.

    For exceptions, the compact format is used by default. So with your example, the noop call in BaseUser::callNoop will raise an Ice::UnknownUserException if BaseUser doesn't know about the derived exception (it can't slice it since it was sent with the compact format). The callNoop invocation will therefore raise UnknownUserException in this scenario.

    If the server hosting the Base interface is configured with Ice.SlicedFormat or if you add the ["format:sliced"] metadata to the Base::noop() definition, the exception will be sent with with the sliced format and the BaseUser::callNoop invocation will be able to Slice the exception to the BaseException type. So the callNoop invocation will raise BaseException in this scenario.

    Finally (and this is new with 3.5.0), you can also add ["preserve-slice"] to the definition of the BaseException exception. If you add this and use the sliced format, the exception will be preserved when un-marshalled in the implementation of BaseUser::callNoop. The callNoop invocation in the client will therefore raise DerivedException.

    To summarize:
    • if compact format is used: the invocation will raise Ice::UnknownUserException
    • if sliced format is used: the invocation will raise BaseException
    • if sliced format is used and BaseException is marked with ["preserve-slice"]: the invocation will raise DerivedException.

    Cheers,
    Benoit.
  • HI Benoit,

    thanks again for your time and the detailed answer. A few remarks:
    benoit wrote: »
    Note that Slice checksums are designed to provide a check for type equality not "on-the-wire" compatibility. So optionals parameters and data members are used in the checksum computation.

    I was under the impression that part of the motivation for optional data members is that it would make upgrading components in a distributed systems easier (so you could add optional members to a class and an older deployment of the software could still use those data structures as long as it's not relying on any of the new optional parameters). Is this a misunderstanding at my end? If not, this means that the feature won't mix with slice checksums. Not the end of the world, but something to consider when deciding how to maintain consistency (originally I thought those optional data members are also available for structs, which is not the case, so this is far less important to us over all).

    I definitely like that 3.5 improves classes considerably (besides Python 3 support the most significant improvement for our use case). Kudos for that.

    A few more practical conclusions and remarks (please correct me if I'm wrong):
    • Within a domain/system, everybody involved should have the same understanding of default settings used (pretty much like it is the case for MaxMessageSize already) - this includes developers and system administrators. Setting Ice.Default.SlicedFormat=1 (or forgetting so) could change the behavior of the system considerably, so it might even make sense to add a check on application start up for applications that rely on it.
    • Using compact type ids should be limited to very specific use cases and requires coordination to avoid identifier clashes.
    • When interconnecting domains/systems one should make very little assumptions about the configuration within the other systems (so don't rely on slicing or compact) unless there is complete control over both domains/systems.
    • I would have preferred that the sliced format stays the default and compact format would have been an option (Ice.Default.CompactFormat=1). This has the potential to break existing applications. At the very least I would suggest to add information on this to Upgrading your Application from Ice 3.4 - Ice 3.5 - ZeroC . I does mention the new encoding in general, but IMHO it should point out explicitly that the new default compact encoding might break applications that rely on slicing (both classes and exceptions).
    benoit wrote: »
    For exceptions, the compact format is used by default. So with your example, the noop call in BaseUser::callNoop will raise an Ice::UnknownUserException if BaseUser doesn't know about the derived exception (it can't slice it since it was sent with the compact format). The callNoop invocation will therefore raise UnknownUserException in this scenario.

    If the server hosting the Base interface is configured with Ice.SlicedFormat or if you add the ["format:sliced"] metadata to the Base::noop() definition, the exception will be sent with with the sliced format and the BaseUser::callNoop invocation will be able to Slice the exception to the BaseException type. So the callNoop invocation will raise BaseException in this scenario.

    Finally (and this is new with 3.5.0), you can also add ["preserve-slice"] to the definition of the BaseException exception. If you add this and use the sliced format, the exception will be preserved when un-marshalled in the implementation of BaseUser::callNoop. The callNoop invocation in the client will therefore raise DerivedException.

    To summarize:
    • if compact format is used: the invocation will raise Ice::UnknownUserException
    • if sliced format is used: the invocation will raise BaseException
    • if sliced format is used and BaseException is marked with ["preserve-slice"]: the invocation will raise DerivedException.

    I'm not entirely sure if I do like this, on the other hand I wasn't aware that "preserve-slice" wasn't the default behavior beforehand, so probably this isn't that important overall. I do like that "preserve-option" is available now.

    Cheers,
    Michael
  • mes
    mes California
    grembo wrote: »
    I would have preferred that the sliced format stays the default and compact format would have been an option (Ice.Default.CompactFormat=1). This has the potential to break existing applications. At the very least I would suggest to add information on this to Upgrading your Application from Ice 3.4 - Ice 3.5 - ZeroC . I does mention the new encoding in general, but IMHO it should point out explicitly that the new default compact encoding might break applications that rely on slicing (both classes and exceptions).
    Good suggestion, I've updated the release notes.

    We considered making the sliced format the default, but ultimately went with the compact format because it's more efficient. We also felt the use cases for the slicing feature were relatively rare. In other words, it's a very useful feature, and some applications may depend on it as part of their design, in which case explicitly activating it via metadata or a configuration property is trivial and makes the dependency more obvious.

    In other cases, applications might "accidentally" rely on the feature in that it prevents a receiver from failing when Slice definitions are out of sync. In this case I think it's better for developers to discover the situation and then decide whether it's a feature or a bug.

    Take care,
    Mark