Another fun SSL issue today. We managed to get request signing working with a self signed certificate (see this post) but once we bought a real certificate from Gandi things stopped working.
Gandi provided us with a public certificate and a private key, as well as a link to their intermediate certificates. They provide SHA1 and SHA2 certificates, so the first thing to do was to work out which we needed.
openssl can help with that
openssl x509 -text -in /path/to/certificate | grep "Signature Algorithm" => Signature Algorithm: sha256WithRSAEncryption
In this case, it’s a SHA2 certificate.
Next, we need to generate a public certificate bundle that identifies our key. We can use
openssl again to make sure that the chain verifies.
cat public.crt | openssl verify => stdin: OU = Domain Control Validated, OU = Gandi Standard SSL, CN = sub.example.com error 20 at 0 depth lookup:unable to get local issuer certificate
As we can see, providing just our public certificate means that it can’t verify the chain. Let’s try adding the intermediate certificates too
cat inter.crt public.crt | openssl verify => stdin: C = FR, ST = Paris, L = Paris, O = Gandi, CN = Gandi Standard SSL CA 2 error 20 at 0 depth lookup:unable to get local issuer certificate
This time we get another error – it couldn’t verify Gandi’s intermediate certificate. We need to find a trusted certificate that was used to sign Gandi’s intermediate certificates. This is provided on Gandi’s intermediate cert page under the “Cross Signed Certificate” header.
$ cat root.crt inter.crt public.crt | openssl verify => stdin: OK
Once we add this, the chain of certificates verifies correctly.
At the moment, my Apache virtualhost looks something like this:
<VirtualHost *:443> DocumentRoot /var/www/site SSLEngine On SSLCertificateFile /etc/ssl/oursite/all.crt SSLCertificateKeyFile /etc/ssl/oursite/all.key SSLVerifyClient optional SSLVerifyDepth 2 SSLOptions +StdEnvVars SSLCACertificateFile /etc/ssl/gandi/ca-chain.pem </VirtualHost>
SSLCACertificateFile option expects a certificate chain to trust, with our public key first and the key that signed it directly after.
We can generate the file by concatenating everything that verified together
cat public.crt inter.crt root.crt > ca-chain.pem
This should contain everything that Apache needs to verify our client certificate. However, when I make a request I get an “unknown ca” alert.
$ curl -v --cert cert --cacert ../ssl/foobar/ca.crt https://ec2-1-2-3-4.compute-1.amazonaws.com * Rebuilt URL to: https://ec2-1-2-3-4.compute-1.amazonaws.com/ * Hostname was NOT found in DNS cache * Trying 220.127.116.11... * Connected to ec2-1-2-3-4.compute-1.amazonaws.com (18.104.22.168) port 443 (#0) * successfully set certificate verify locations: * CAfile: ../ssl/oursite/ca.crt CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS alert, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server key exchange (12): * SSLv3, TLS handshake, Request CERT (13): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS handshake, CERT verify (15): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS alert, Server hello (2): * error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca * Closing connection 0 curl: (35) error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
I wasn’t sure if this was an issue on the client or server side to start with as I’m using a self signed certificate on the server. However, if I tried the request without the
--cert option it worked fine. I’ve since confirmed that this error is the server rejecting the client certificate because of an unknown CA
But, how is this possible? We ran everything through
openssl and it verified fine. After many hours of searching and trying various things (yes, the ca-chain.pem file needs to be public at the top, root CA at the bottom, definitely) we enabled the Apache debug log by adding
LogLevel debug to our Virtualhost.
After restarting Apache and running
tail -f /var/log/apache2/error.log, we noticed the following error appearing:
[Wed Jun 17 11:34:27.181869 2015] [ssl:info] [pid 6112] [client 22.214.171.124:44048] AH02276: Certificate Verification: Error (20): unable to get local issuer certificate [subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US / issuer: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE / serial: 13EA28705BF4ECED0C36630980614336 / notbefore: May 30 10:48:38 2000 GMT / notafter: May 30 10:48:38 2020 GMT
It looks as though the UserTrust certificate that Gandi provide isn’t being trusted, and Apache is looking for the AddTrust CA Certificate. Searching Google for “AddTrust CA root” lead me to a comodo knowledgebase page which provides the certificate.
ca-chain.pem and restarting Apache worked, and once I’d restarted Apache I could make my request fine.
The AddTrust CA certificate can replace the UserTrust certificate, so to generate a complete CA chain for certificates signed by Gandi, you need the following certificates in order:
- Your public cert
- Gandi intermediate certificates (SHA2)
- AddTrust cert
You can verify that all of the correct CA certificates are loaded with the following command (where
combined.pem is your private key and public cert concatenated):
openssl s_client -cert /path/to/combined.pem -connect ec2-1-2-3-4.compute-1.amazonaws.com:443
There should be a section in the response that contains the following:
Acceptable client certificate CA names /C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2 /OU=Domain Control Validated/OU=Gandi Standard SSL/CN=sub.example.com /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority /C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
If you’re missing any of them, double check
ca-chain.pem and make sure that all the certificates that you’re expecting are in there.
Michael is a polyglot software engineer, committed to reducing complexity in systems and making them more predictable. Working with a variety of languages and tools, he shares his technical expertise to audiences all around the world at user groups and conferences. You can follow @mheap on Twitter