ZeroC Ice running in docker container - SSL issue

chrisg

Hi all,

I am testing running a client in a docker container which connects to a remote server component via glacier. It works perfectly on physical hosts or a virtual box vm. When running in a docker container I have found after a long running session (several days) fails the client is unable to reconnect. The clients std.out gives me this message whenever this occurs:

OpenSSLTransceiverI.cpp:481: Ice::ConnectionLostException: connection lost: recv() returned zero

I set Ice.Trace.Security=1 and the following extra information is provided via std.err with this occurs:

-- 06/27/17 06:58:32.272 /data/test/client: Security: certificate verification failure
issuer = /C=US/O=GeoTrust Inc./OU=(c) 2008 GeoTrust Inc. - For authorized use only/CN=GeoTrust Primary Certification Authority - G3
subject = /C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G2
depth = 1
error = unable to get local issuer certificate
local address = ********:47708
remote address = ********:6058

Any time this occurs a quick restart of docker container resolves.

Any thoughts or guidance would be much appreciated.

Cheers,
Chris

joegeorge

Hi @chrisg,

A few questions:
1. What operating system and Ice version are you using in your Docker container?
2. Do you have multiple endpoints configured? For instance tcp and ssl endpoints.
3. Does your client certificate bundle contain the GeoTrust intermediate certificate?

It's not completely clear what's happening from your description. Can you also provide the the log with Ice.Trace.Network=3 set around the establishment of the initial working connection as well as the failure?

chrisg

Hi @joegeorge

Thank you for quick response. Answers as follows:
1. Container is Ubuntu 14.04, Ice 3.6.3.
2. There is only one ssl endpoint configured.
3. To simplify deployment I'm using IceSSL.UsePlatformCAs=1. The docker image I am using installs 'ca-certificates' package which has the CA certs it requires to perform SSL handshake. Indeed it does appear the first time the session is started when the container boots that it can find the CA, it is only on subsequent reconnects after the container/client has been running for sometime that it doesn't seem to be able to find the CA certificate.

I'll put network trace on and see what I see. I'll also run a test where I explicitly declare the CA certificate in client certificate bundle and see if this alleviates the issue.

Will report findings back.

Cheers
Chris

chrisg

Hi @joegeorge,

Your questions prompted some thought, particularly number 2.

It is actually caused by use of two different communicators (in different threads of the same application) with two different IceSSL configurations.

When I look at the start order of the client application it starts the thread with remote communicator first and then the communicator with local endpoint and self signed IceSSL configuration. Due to the way our application was managing it's Ice::Properties it was then overriding the IceSSL configuration on the remote communicator so when it try's to reconnect to glacier later it is using the wrong IceSSL configuration which results in the IceSSL certificate errors. I verified this by disabling SSL on the local endpoint and it works as expected on subsequent reconnects.

End of story, it was application error which has been resolved.

Thanks for your help.

Cheers,
Chris

joegeorge

Glad to hear you solved the issue, @chrisg!