Extending Glacier2 to allow customized request filtering

grembo

Hello,

we're currently evaluating a setup, where a Glacier2 router is used as the single endpoint for user sessions to provide a clear separation of front-end and backend code. Since we want to make sure, that we can do session based access control to various objects, we looked into the options Glacier2 provides out of the box (filter by Category, Identity, AdapterId etc.).

Unfortunately these options don't quite cut it in our case, since we would like to also grant access based on operations and sometimes even based on parameters - in the past we would have written a specialized facade to do the fine grained locking, but this means more maintenance and slows down development efforts. More importantly, we would really like to have a situation where the Glacier2 admin can setup filter rules which then can be audited outside of the scope of a normal code audit (pretty much like what's usually done with IP firewalls).

I checked the Glacier2 code and it seems like that object identity based filters are currently evaluated in ClientBlobject.cpp (Glacier2::ClientBlobject::ice_invoke_async). Altering this function should be really straightforward to customize it for our needs, but obviously I don't want to maintain our own bespoken version of Glacier2.

So - since fine grained access control can be extremely use case specific and I'm pretty certain that our needs differ from those of others - I would like to modify this code to allow the inclusion of plugins into the filter chain by configuration. This way the code change could be useful for many different projects/scenarios.

I would suggest something like Ice.Plugin.Glacier2ClientFilter (if useful, Ice.Plugin.Glacier2ServerFilter could be included into ServerBlobject later as well).

So, do you

1. think this design makes sense?
2. plan on implementing anything like this on your own?
3. and if not, would you be willing to include such code into your codebase (assuming it matches your expectations in terms of code quality etc.)?

Thanks,
Michael

mes

Hi Michael,

To date we haven't had any requests for finer-grained filtering in Glacier2, so at this time we don't have plans to add such a feature.

Having said that, it seems like a reasonable idea. There are a couple of things to keep in mind:

• You won't have access to the operation parameters unless you manually extract them using the stream API.
• Filtering based on operation names can be problematic. For example, two different interfaces might have operations with the same name. Nothing in the request specifies which interface is being used, but you might able to infer it based on the object identity or by passing some custom context value.

Regards,
Mark

grembo

First version available

Hi Mark,

I have a first working version of the patch - including a proof of concept unit test - which I would like to discuss. Would you like to do this publicly in the ZeroC forums or would you prefer to discuss this via e-mail first? I would like to give you a more detailed overview of what I did and a rationale why I did certain things like I did and how this could be useful to other users as well - and hopefully get some feedback on how to improve this so it meets your standards.

Cheers,
Michael

grembo

General idea

Hi Mark,

As Bernard pointed out yesterday, ZeroC is very short on time and needs to focus on implementing features for paying customers. I do respect this, but at the same time - with your permission - will continue posting in this thread about this patch, since even if you don't consider this useful, somebody else might benefit from this.

----

General idea:

The Glacier2 router is used as a point of separating different layers of the system. In our case the client is a Python based web page running in the DMZ, while the backend Ice services are running in a private network behind a second firewall utilizing IceGrid. All connections between grid nodes, Glacier2 and python clients are secured using SSL and client certificates.

It's important to note that user sessions won't match Glacier2 sessions in this case, instead a Glacier2 session just authorizes the connection between the web process and the router, which then connects to the persistent per user management behind the router and passes the per user session identifier in the context.

Services within the grid are managing valuable data, which should not leave the system unless authorized. For that reason all external traffic has to traverse the Glacier2 router. The feature the router is missing so far to make this work is more sophisticated service level firewalling/filtering technology, which is introduced by this patch. Since filtering is a complex and application specific task, we decided to use a plugin approach, to allow different filter algorithms depending on the exact requirements. It should also be possible to load various plugins (in a defined order when necessary), so filters can be combined by configuration.

Filtering should not only be done incoming, but also for outgoing traffic. Examples for data to be filtered are:

• Only allow incoming calls that contain no proxies in paramters (and therefore systematically prevent proxy "spoofing")
• Only allow outgoing calls that contain no proxies in outparams
• Only allow a certain set of operations in incoming calls
• Limit calls to a well defined list of proxies (defined by facet / object identifier / adapter id) in combination with certain operations
• Only allow calls to certain proxies, that are allowed from within the session passed in the context (prevents the loss of huge amount of data in case the web server is compromised)
• Filter outgoing responses for certain information - e.g. automatically scan for credit card numbers in strings and block outgoing data if this occurs (including logging)
• Filter outgoing exceptions (not just UserExceptions) as debug information in exceptions might accidentally reveal secret information
• Simple provide detailed logging of accesses / enable accounting or rate limiting of calls

So overall, the filter plugins applied will be relatively small and could be application specific or - depending on the implementation - firewall like with a flexible configuration syntax. The important part is, that all traffic between different security levels of the systems goes through a well defined Ice router/firewall, and the filter logic is contained in one simple module that is easy to audit as part of regulation.

So if you're subject to industry security standards that require you to proof measures taken to segregate systems (e.g. PCI DSS and handling of PAN data), it's important to be able to audit the measures taken to prevent accidental as well as intentional leakage of information. In this case a combination of filtering incoming traffic (access control) and filtering outgoing traffic for information that shouldn't leave the system, Implementing a firewall at the Ice protocol level seems like a straight forward solution to accomplish this task.

Once a plugin structure is established, plugin implementations themselves can be run as separate projects that are pushed forward independently of the Ice core.

grembo

Filter plugin definition

Glacier2::FilterPlugin inherits from Ice::Plugin and provides the following methods:

virtual std::string getName() = 0;
To be overridden by a plugin specialization, return a meaningful identifier.

Functions to filter calls initiated by clients:

MatchResult matchClientIncoming(...)
Match an inbound call from a Glacier2 client.

MatchResult matchClientOutgoing(...)
Match an outbound call to a Glacier2 client (read: filter an Ice response), this includes UserExceptions as well.

MatchResult matchClientOutgoingException(...)
Match an outbound exception (this only affects system exceptions inherited from Ice::Exception)

Functions to filter calls initiated by Glacier2 (client side server callbacks aka reverse connection):

MatchResult matchServerOutgoing(...)
Match an outbound call to a server callback.

MatchResult matchServerIncoming(...)
Match an inbound call from a server callback (read: filter an Ice response given by a client in a reverse connection), this includes UserExceptions as well.

MatchResult matchServerIncomingException(...)
Match an inbound exception (this only affects system exceptions inherited from Ice::Exception)

MatchResult is an enum providing the following values:

• MatchIgnore: Ignore this rule (default for all functions)
• MatchAccept: Rule accepts this call (but others might deny it)
• MatchAcceptQuick: Accept this rule immediately, stop evaluating any other rules.
• MatchDeny: Deny this rule (has no effect if another rule
• MatchDenyQuick: Deny this rule immediately, stop evaluating any other rules.

Note: The exact logic of evaluating these rules has been inspired by the existing filter rules, it might be worthwhile to reconsider if this really makes sense or if it would be better to decouple the two mechanisms.

Evaluation logic:

1. Accept a call if there are no filters configured
2. Deny a call, if filters are configured and there is no single MatchAccept(Quick) response or at least one MatchDenyQuick.

(MatchIgnore rules don't count as configured filters).

On top of this, all match calls can throw exceptions on their own to end evaluation of further rules and pass on a specific local exceptions. By default an ObjectNotFoundException will be emitted (containing __FILE__ and __LINE__) by the filter code.

The full header file is located in include/Glacier2/FilterPlugin.h and looks something like this:

#ifndef FILTER_PLUGIN_H
#define FILTER_PLUGIN_H

#include <string>
#include <vector>
#include <Ice/Plugin.h>
#include <IceUtil/Handle.h>

namespace Glacier2
{
    class FilterPlugin : virtual public Ice::Plugin
    {
    public:
        // get filter name
        virtual std::string getName() = 0;

        enum MatchResult
            {
                MatchIgnore,      // ignore result (e.g. for logging)
                MatchAccept,      // accept call (might be changed by a later rule)
                MatchAcceptQuick, // accept call and stop processing further rules
                MatchDeny,        // deny call (might be changed by a later rule)
                MatchDenyQuick    // deny call and stop processing further rule
            };

        // this is called on inbound calls by clients
        virtual MatchResult matchClientIncoming(const Ice::ObjectPrx&, const std::pair<const Ice::Byte*, const Ice::Byte*>&, 
                                                const Ice::Current&, const Ice::Context&)
        {
            return MatchIgnore;
        }

        // this is called on outbound calls to clients (= return). bool parameter determines if the call
        // succeeded or a user exception occured
        virtual MatchResult matchClientOutgoing(const Ice::ObjectPrx&, bool, const std::pair<const Ice::Byte*, const Ice::Byte*>&, 
                                                const Ice::Current&, const Ice::Context&)
        {
            return MatchIgnore;
        }

        // this is called on local exceptions returned to clients, user exceptions are handled
        // in matchClientOutgoing
        virtual MatchResult matchClientOutgoingException(const Ice::ObjectPrx&, const Ice::Exception&, 
                                                         const Ice::Current&, const Ice::Context&)
        {
            return MatchIgnore;
        }

        // this is called on outbound calls (callbacks) to servers / reverse connections
        virtual MatchResult matchServerOutgoing(const Ice::ObjectPrx&, const std::pair<const Ice::Byte*, const Ice::Byte*>&, 
                                                const Ice::Current&, const Ice::Context&)
        {
            return MatchIgnore;
        }

        // this is called on inbound calls from servers / reverse connections (= return). bool parameter
        // determines if the call succeeded or a user exception occured
        virtual MatchResult matchServerIncoming(const Ice::ObjectPrx&, bool, const std::pair<const Ice::Byte*, const Ice::Byte*>&, 
                                                const Ice::Current&, const Ice::Context&)
        {
            return MatchIgnore;
        }

        // this is called on local exceptions returned by servers / reverse connections, user exceptions 
        // are handled in matchServerIncoming
        virtual MatchResult matchServerIncomingException(const Ice::ObjectPrx&, const Ice::Exception&, 
                                                         const Ice::Current&, const Ice::Context&)
        {
            return MatchIgnore;
        }


        // defaults for Ice::Plugin
        virtual void initialize() { }
        virtual void destroy() { }
    };

    typedef IceUtil::Handle<FilterPlugin> FilterPluginPtr;
    typedef std::vector<FilterPluginPtr> FilterPlugins;
}

#endif

grembo

Changes to Glacier2 and request for feedback

The following changes need to be done, to make this feature work in Glacier2:

• Adapt FilterManager to locate configured plugins (using the standard Ice plugin mechanism) and provide functions to access plugins (vector of FilterPluginPtr)
• Move some variables from ClientBlobject to Blobject, since they're required by both specializations now (FilterManager, RejectTracelevel etc.). Adapt Client- and ServerBlobject constructors to allow passing of those values.
• Enable SSL context forwarding server side (this feature has been disabled for reasons I don't understand, there are configuration variables in the code for it, but it has never been enabled). This is not strictly required to make this work, but seemed to be a reasonable thing to do.
• Evaluate (all) matchClientIncoming in ClientBlobject::ice_invoke_async, make it interact nicely with the other filter settings. It might be better to make this mechanism completely independent of the existing filter mechanisms - they're interacting nicely, but the existing filter code feels kind of incomplete and might give a false sense of security.
• Evaluate (all) matchServerOutgoing in ServerBlobject::ice_invoke_async
• Extend RequestQueue and Request to evaluate (all) matchClientOutgoing OR matchServerIncoming as well as matchClientOutgoingException OR matchServerIncomingException (depending on the direction/type of the call). Affects Request::response and Request::exception and the calling RequestQueue counterparts.
• Extend AMI_Array_Object_ice_invokeTwowayI so that ice_response evaluates (all) matchClientOutgoing OR matchServerIncoming and ice_exception evaluates (all) matchClientOutgoingException OR matchServerIncomingException (depending on the direction/type of the call). This involves passing a couple of more object references and a copy of current and context to the AMI callback object - this happens on every call right now, even if there are no filters configured. Without benchmarking it is not clear, if this will has a significant performance impact on Glacier2 - given the code structure it shouldn't make a huge difference, but it's something to be discussed. Maybe it would make more sense to do some optimization here at the cost of duplicating some code.
• Extend AMI_Array_Object_ice_invokeOnewayI in Blobject.cpp so ice_exception evaluates (all) matchClientOutgoingException OR matchServerIncomingException (depending on the direction/type of the call). Even though oneway calls should not leak any information, this can be useful for a logging plugin this should be discussed, maybe this is just overkill.

Request for feedback
The current version of the patch - excluding unit tests - is pretty compact and already fully functional. There's still potential to make the code cleaner, which will hopefully happen over the next few weeks and we will test it in staging as well as production to make sure it delivers before I will post a patch in the public forum. Nevertheless, it would be nice to receive some feedback on what I've posted so far, maybe somebody can think of more use cases or extensions to make this applicable to a broader range of setups/businesses - it's clearly not for everybody, but for people handling sensitive data this could be a very useful addition to their security strategy.

If anybody is interested in testing the patch themselves feel free to PM me and I will provide a copy.

--
Michael