Archived

This forum has been archived. Please start a new discussion on GitHub.

Connection to server blocks when made in a CertificateVerifier

Hi everyone ! Me again...

I am facing a very problematic problem. :)
Here is the situation:
I would like to implements the IceSSL plug-in in our product in order to securize the connections between each processes.

After studying the Ice documentation, I managed to establish a development plan to integrate a CRL (Certificate Revocation List) verification.

To do so, I made a very simple server that read the CRL file and gives copies of it to the processes that request it via a TCP connection.
The request is made in a plug-in (with Ice::Plug-in interface) that adds to the IceSSL plug-in a CertificateVerifier object (extends IceSSL::CertificateVerifier).
The CV holds a CommunicatorPtr that holds the process communicator (there is only one of it), a CRLGetter proxy (to the CRL Server). When the "verify" function of CV is called, if the proxy is not initialized, it attempts to connect to the CRL server.

I have 2 processes for testing this CV plug-in. One is server (it holds the object that fits the Slice interface), the other is the client, invoking some operations.
When I define the server endpoint with TCP, ok no problem. When I define the endpoint with SSL but WITHOUT using my CV, ok no problem.
BUT when I use SSL with my CV plug-in, the server is ok, ie. it connects to the CRL server, gets the CRL and validates the certificate. However, the client blocks on connection establishment... The "verify" is called, it detects well that the proxy is not initialized, an ObjectPrx is created successfully by the communicator and finally it blocks when it has to cast the ObjectPrx to a CRLServerPrx (with checkedCast, but with uncheckedCast it does the same).

I used Ice.Trace.Network/Protocol and I only have (on the client side):
-- 08/03/12 17:50:57.882 Network: trying to establish ssl connection to 192.168.24.109:11000
-- 08/03/12 17:50:58.005 Network: trying to establish tcp connection to 192.168.24.109:10100
With:
Ice.Trace.Network=2
Ice.Trace.Protocol=2
(Where 192.168.24.109 is my IP, it is the same with 127.0.0.1)

I think I exposed all the informations that are at my disposal...
However, if you need something more, ask me.

Thanks to read me !
Daniel

Edit: Maybe that would be interesting: the checkedCast seems to block when IceInternal::checkedCastImpl calls "ice_isA" or "ice_staticId" (lines 898-899 of Proxy.h)

Comments

  • mes
    mes California
    Hi Daniel,

    I suspect you have a thread starvation issue here. If I understand things correctly, the SSL endpoint 11000 is your application server, and the TCP endpoint 10100 is your CV server. The client attempts to establish an SSL connection; before the connection is fully established, the client must first validate the server's certificate. IceSSL invokes your plug-in from a thread in the Ice client-side thread pool; your plug-in then attempts to invoke on the CV server.

    The default configuration of the Ice client-side thread pool allows a maximum of only one thread. If this thread is blocked in a nested invocation, no thread is available to process the nested invocation's reply.

    Try increasing the maximum size of the client-side thread pool to see if I'm right. You can do this by setting the following property:

    Ice.ThreadPool.Client.SizeMax=2

    Regards,
    Mark
  • You were right Mark.
    However I set
    Ice.ThreadPool.Client.Size=2
    
    and not
    Ice.ThreadPool.Client.SizeMax=2
    
    (Don't work in that case)

    Thank you.

    Daniel
  • CertificateVerifier returns false, but connection continue

    Another problem...
    I think I missed something... But I can't find what !

    My CertificateVerifier acts correctly and return false on an invalid certificate (ie. revoked certificate)

    However, when the client application verifies the server identity (that is invalid in my example), is there any way that I can be aware of this rejection ?
    For now, the client continues its execution even if the certificate verifier returned false...
  • mes
    mes California
    IceSSL raises a SecurityException if a verifier returns false (see the implementation of IceSSL::Instance::verifyPeer in cpp/src/IceSSL/Instance.cpp).

    Are you saying that the client's connection is established even though the verifier returned false?

    If so, please describe your configuration (Ice version, compiler version, OS version). It would also be helpful to see the log output of your client with the following properties defined:

    Ice.Trace.Network=2
    Ice.Trace.Protocol=1
    IceSSL.Trace.Security=1

    Regards,
    Mark
  • Yes, I saw in the source code that an exception is raised, but it is never returned to the main client code...
    I use Ice 3.4.2 with Visual C++ compiler v15.00.30729.01 (Visual Studio 2008 SP1) on Windows 7 x64 (however I develop 32bits software)

    You can find attached the log of the client and server sides.
    Be careful, there is some of my traces, but you will recognize them easily.

    The certificate verifier is installed by a custom plug-in in the IceSSL plug-in. The IceSSL plug-in is initialized before the verifier installer. (I followed the documentation to do that)

    Like I presented in my first post, the certificate verifier contacts a CRL distributor in TCP, then verifies that the peer certificate is not in the CRL.
    In my configuration, my client has a revoked certificate, and it is well detected by the CV. But the connection is still established and no exception are raised in the main client process (when I "checkedCast" the proxy).

    Waiting for your answer.

    Daniel
  • mes
    mes California
    Hi,

    I wrote a small test case to try to reproduce this problem, but the code works as expected. My certificate verifier's implementation of verify returns false, which causes a SecurityException to be raised to the client.

    If you're still having trouble, the quickest way to resolve it is for you to provide a small test case that I can use to reproduce the problem.

    Regards,
    Mark
  • Hi,

    Well... There is something strange. I corrected some code in my verifier and now the connection is well refused. I will double-check what I modified to find something relevant. I will post it here if I find.
    For now, I can't reproduce the problem I had yesterday.

    I tested the new CV with an invalid certificate used by the server, it causes a Security Exception on the client side. Then the client used the invalid certificate and that causes a Connection Lost Exception (with the reason: "recv() returned 0") on the client.

    Finally, there is a point that I am unable to explain: when the client attempts to connect to the server, it seems there are 2 connection attempts (I can see 2 calls to the CV). Is it normal ? Is it configurable ?

    Thanks.
    Daniel
  • Hi,

    Some investigation (and "brain heating") later, I found why the connection was accepted before my modifications.
    That's almost all my fault.
    First, notice that there is 2 connection attempts.
    On the first one, my CV rejected it successfully. But, on the second one, due to an implementation mistake, my CV accepted it. So the connection was established.
    The correction that I made today corrected the mistake.

    So why almost ? Because I didn't know (or even notice) that there were 2 connections attempts... So I ask the same questions that are in my previous post.

    Finally, I would express my opinion on the IceSSL implementation. Despite the lack of a CRL support implementation (due to the lack of standard), this plug-in provides enough ressources that enables anyone to do a customized implementation of a CRL, quite easily. And that is a very good point. Also, to let an access to the low-layer OpenSSL stuff is also a very good idea. The only major problem that I encoutered is the lack of tutorials on OpenSSL functionalities. But it's not the ZeroC repsonsability to provide such a thing.

    Thank you Mark for your help !
    Daniel
  • mes
    mes California
    Dany2901 wrote: »
    Finally, there is a point that I am unable to explain: when the client attempts to connect to the server, it seems there are 2 connection attempts (I can see 2 calls to the CV). Is it normal ? Is it configurable ?
    Yes, the two connection attempts are normal, and yes, the behavior is configurable. This is Ice's automatic retry feature.

    After further investigation, we do not recommend making remote invocations from within a certificate verifier's verify method. The problem is that the verifier is invoked during the connection establishment process, so using a connection here can cause several issues, such as interfering with the non-blocking guarantee of AMI invocations, deadlocks, and even a crash.

    We will do something to resolve this in the next release, but we haven't decided exactly what we'll do yet. :) It could be that we simply document that remote invocations should not be made by a verifier, or we might make some changes to the code to make this possible.

    My colleague Benoit suggested a potential workaround: your plug-in can use a background thread to periodically retrieve the CRL, in which case your verifier only needs to consult this local list and avoids the remote invocation.

    Regards,
    Mark
  • Thank you for your answer.

    I take note of your warning. I think Benoit's idea is a good one (and not so difficult to implement :)).
    Everything is working fine now ! Again, thank you for your help.

    Concerning remote invocations in a verifier, if a future release of Ice safely allows to make remote invocations in a verifier, I think it would be a good idea to, firstly, document explicitly that is a possibility and, secondly, insist on that, conceptually, a verifier can't establish a secure (ie. SSL/TLS) connection to another peer because it is verifying another one, so in most cases the connection that is used is unsecured. Aren't you agree ?

    Best regards,
    Daniel
  • mes
    mes California
    Dany2901 wrote: »
    Concerning remote invocations in a verifier, if a future release of Ice safely allows to make remote invocations in a verifier, I think it would be a good idea to, firstly, document explicitly that is a possibility and, secondly, insist on that, conceptually, a verifier can't establish a secure (ie. SSL/TLS) connection to another peer because it is verifying another one, so in most cases the connection that is used is unsecured. Aren't you agree ?
    Yes, I agree.

    Regards,
    Mark