#ssl

curl: (35) error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca

17 Jun 2015 in Tech

Another fun SSL issue today. We managed to get request signing working with a self signed certificate (see this post) but once we bought a real certificate from Gandi things stopped working.

Gandi provided us with a public certificate and a private key, as well as a link to their intermediate certificates. They provide SHA1 and SHA2 certificates, so the first thing to do was to work out which we needed. openssl can help with that

bash
openssl x509 -text -in /path/to/certificate | grep "Signature Algorithm"
=> Signature Algorithm: sha256WithRSAEncryption

In this case, it's a SHA2 certificate.

Next, we need to generate a public certificate bundle that identifies our key. We can use openssl again to make sure that the chain verifies.

bash
$ cat public.crt | openssl verify
=> stdin: OU = Domain Control Validated, OU = Gandi Standard SSL, CN = sub.example.com
    error 20 at 0 depth lookup:unable to get local issuer certificate

As we can see, providing just our public certificate means that it can't verify the chain. Let's try adding the intermediate certificates too

bash
$ cat inter.crt public.crt | openssl verify
=> stdin: C = FR, ST = Paris, L = Paris, O = Gandi, CN = Gandi Standard SSL CA 2
    error 20 at 0 depth lookup:unable to get local issuer certificate
```
This time we get another error - it couldn't verify Gandi's intermediate certificate. We need to find a trusted certificate that was used to sign Gandi's intermediate certificates. This is provided on Gandi's intermediate cert page under the "Cross Signed Certificate" header.
```bash
$ cat root.crt inter.crt public.crt | openssl verify
=> stdin: OK

Once we add this, the chain of certificates verifies correctly.

At the moment, my Apache virtualhost looks something like this:

<VirtualHost *:443>
    DocumentRoot /var/www/site

    SSLEngine On
    SSLCertificateFile /etc/ssl/oursite/all.crt
    SSLCertificateKeyFile /etc/ssl/oursite/all.key

    SSLVerifyClient optional
    SSLVerifyDepth 2
    SSLOptions +StdEnvVars
    SSLCACertificateFile /etc/ssl/gandi/ca-chain.pem
</VirtualHost>

The SSLCACertificateFile option expects a certificate chain to trust, with our public key first and the key that signed it directly after.

We can generate the file by concatenating everything that verified together

bash
cat public.crt inter.crt root.crt > ca-chain.pem

This should contain everything that Apache needs to verify our client certificate. However, when I make a request I get an "unknown ca" alert.

bash
$ curl -v --cert cert --cacert ../ssl/foobar/ca.crt  https://ec2-1-2-3-4.compute-1.amazonaws.com
    * Rebuilt URL to: https://ec2-1-2-3-4.compute-1.amazonaws.com/
    * Hostname was NOT found in DNS cache
    *   Trying 1.2.3.4...
    * Connected to ec2-1-2-3-4.compute-1.amazonaws.com (1.2.3.4) port 443 (#0)
    * successfully set certificate verify locations:
    *   CAfile: ../ssl/oursite/ca.crt
      CApath: /etc/ssl/certs
    * SSLv3, TLS handshake, Client hello (1):
    * SSLv3, TLS alert, Client hello (1):
    * SSLv3, TLS handshake, Server hello (2):
    * SSLv3, TLS handshake, CERT (11):
    * SSLv3, TLS handshake, Server key exchange (12):
    * SSLv3, TLS handshake, Request CERT (13):
    * SSLv3, TLS handshake, Server finished (14):
    * SSLv3, TLS handshake, CERT (11):
    * SSLv3, TLS handshake, Client key exchange (16):
    * SSLv3, TLS handshake, CERT verify (15):
    * SSLv3, TLS change cipher, Client hello (1):
    * SSLv3, TLS handshake, Finished (20):
    * SSLv3, TLS alert, Server hello (2):
    * error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
    * Closing connection 0
    curl: (35) error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca

I wasn't sure if this was an issue on the client or server side to start with as I'm using a self signed certificate on the server. However, if I tried the request without the --cert option it worked fine. I've since confirmed that this error is the server rejecting the client certificate because of an unknown CA

But, how is this possible? We ran everything through openssl and it verified fine. After many hours of searching and trying various things (yes, the ca-chain.pem file needs to be public at the top, root CA at the bottom, definitely) we enabled the Apache debug log by adding LogLevel debug to our Virtualhost.

After restarting Apache and running tail -f /var/log/apache2/error.log, we noticed the following error appearing:

[Wed Jun 17 11:34:27.181869 2015] [ssl:info] [pid 6112] [client 134.225.2.2:44048] AH02276: Certificate Verification: Error (20): unable to get local issuer certificate [subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US / issuer: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE / serial: 13EA28705BF4ECED0C36630980614336 / notbefore: May 30 10:48:38 2000 GMT / notafter: May 30 10:48:38 2020 GMT

It looks as though the UserTrust certificate that Gandi provide isn't being trusted, and Apache is looking for the AddTrust CA Certificate. Searching Google for "AddTrust CA root" lead me to a comodo knowledgebase page which provides the certificate.

Appending addtrustexternalcaroot.crt to ca-chain.pem and restarting Apache worked, and once I'd restarted Apache I could make my request fine.

The AddTrust CA certificate can replace the UserTrust certificate, so to generate a complete CA chain for certificates signed by Gandi, you need the following certificates in order:

  • Your public cert
  • Gandi intermediate certificates (SHA2)
  • AddTrust cert

You can verify that all of the correct CA certificates are loaded with the following command (where combined.pem is your private key and public cert concatenated):

bash
openssl s_client -cert /path/to/combined.pem -connect ec2-1-2-3-4.compute-1.amazonaws.com:443

There should be a section in the response that contains the following:

Acceptable client certificate CA names
/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
/OU=Domain Control Validated/OU=Gandi Standard SSL/CN=sub.example.com
/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

If you're missing any of them, double check ca-chain.pem and make sure that all the certificates that you're expecting are in there.