Incomplete chain, and other SSL errors nobody enjoys

"Works in the browser, fails in curl" is not a paradox. Browsers quietly repair broken certificate chains, and nothing else does.

What a chain is, and who sends what

A TLS certificate is only trusted because something else vouches for it. Your leaf certificate for api.example.com was signed by an intermediate CA, and that intermediate was signed by a root CA whose certificate ships in the trust store of the operating system, the browser or the runtime. Validation walks that path: leaf, intermediate, root, stop when you reach something already trusted.

The split exists to protect the root. Root private keys live offline in hardware, are used a handful of times a year, and their certificates are valid for decades. Intermediates do the day-to-day signing and can be replaced without touching the trust store of every device on earth. Let's Encrypt alone has rotated through R3, then R10 and R11, without anyone noticing.

The division of labour is where things break. The client brings the root. The server must send the leaf and every intermediate between them. Send only the leaf and a strict client sees a certificate signed by an authority it has never heard of, with no way to connect it to the root sitting in its trust store. That is the incomplete chain, and it is a server misconfiguration reported as a client error.

One bug, five error messages

The same missing intermediate produces completely different text depending on who noticed it, which is why the problem gets diagnosed five separate times in one organisation.

Two neighbours are worth separating from it. self signed certificate in certificate chain (verify error 19) means the chain ended at a certificate that signed itself and is not in your trust store, which in a corporate network almost always means a TLS-inspecting proxy is re-signing traffic with its own root. And certificate has expired (error 10) may be about the intermediate rather than the leaf, which is a genuinely confusing morning when your own certificate has ninety days left on it.

Why it works in Chrome and nowhere else

Browsers do two things no command-line client does.

They cache intermediates. Once Chrome or Firefox has seen an intermediate on any site, it keeps it and will reuse it to complete a chain later. So a server that has never sent its intermediate looks fine to a developer whose browser picked the certificate up from a different site last week, and broken to a fresh CI container.

They fetch what is missing. X.509 certificates carry an Authority Information Access extension with a caIssuers URL pointing at the issuing certificate. Chrome and Safari follow it, Firefox both preloads a large set of known intermediates and fetches when needed. curl, OpenSSL, Java, Python and Go do not fetch, by design: silently making a network request in the middle of certificate validation is a fingerprinting and availability problem.

The practical consequence is that a browser is the worst possible tool for checking whether your chain is correct. Test with openssl s_client or a fresh container, and read what the server actually sent rather than what a client managed to reconstruct. Pasting the PEM blocks into our SSL certificate decoder does the reading part for you: it matches issuer to subject across the blocks and labels each certificate leaf, intermediate or root, so a missing link in the middle is visible rather than inferred.

Order, fullchain and the config line

Most incomplete chains come from one config line. Certbot writes several files, and it is easy to point at the wrong one:

• cert.pem is the leaf alone. This is the one that breaks things.
• chain.pem is the intermediates alone.
• fullchain.pem is leaf plus intermediates, which is what a server should send.
• privkey.pem is the private key, which never leaves the server and never belongs in a paste buffer.

nginx's ssl_certificate wants fullchain.pem. Apache's SSLCertificateFile has accepted the concatenated form since 2.4.8, which retired SSLCertificateChainFile. HAProxy wants one file containing leaf, intermediates and key together. Cloud load balancers usually take the leaf and the chain in separate fields, and the chain field is the one people leave empty.

Order inside the file is leaf first, then each intermediate that signed the one before it. TLS 1.2 required that order strictly. RFC 8446 kept leaf-first as a requirement for TLS 1.3 and relaxed the rest to a recommendation, on the grounds that implementations already had to tolerate disordered chains in the wild. Take the relaxation as a safety net, not a licence: OpenSSL will build a path from a jumbled bag of certificates, older Java and several embedded stacks will not.

Expired roots and the cross-signing puzzle

Leaf certificates expire on a schedule everyone monitors. Roots expire too, once every couple of decades, and nobody has a dashboard for that.

The case everyone remembers is DST Root CA X3, which expired on 30 September 2021. Let's Encrypt had spent years cross-signing its own ISRG Root X1 with that older root so that devices predating ISRG would still trust it. When it expired, current browsers noticed nothing, and OpenSSL 1.0.2 and Android below 7.1.1 broke instantly. OpenSSL 1.0.2 had a specific defect here: it rejected the whole chain when it found an expired certificate in it, even though an alternative valid path existed.

What follows from that is operational. Monitor the expiry of every certificate in the chain, not just the leaf, and know which root your CA currently chains to. Certificates have also been getting shorter: the CA/Browser Forum cut maximum leaf validity to 398 days in September 2020, and the industry has agreed to keep shortening it, which makes automated renewal the only sane option and makes a monitoring gap much more expensive.

Adjacent and worth checking once: the CAA records on your domain decide which CA is allowed to issue for it at all. A renewal that suddenly fails with an authorisation error, on a chain that was fine yesterday, is often a CAA record that no longer lists the CA you switched to. Our DNS lookup queries CAA directly, which is faster than reasoning about why an ACME challenge failed.

Three commands that find it

The first shows what the server sends:

openssl s_client -connect example.com:443 -servername example.com -showcerts

Read the list at the top. Each entry has a subject (s:) and an issuer (i:), and each issuer should appear as the subject of the next entry. One certificate only, with an issuer that is not a root you recognise, is your answer. The -servername flag is not optional on shared hosting: without SNI the server sends whatever certificate is on the default virtual host, and you will spend twenty minutes debugging the wrong certificate.

The second checks a chain before you deploy it:

openssl verify -untrusted chain.pem cert.pem

This validates offline against your local trust store, so it works in CI and tells you whether the bundle your CA sent is complete.

The third is the one-line smoke test:

curl -vI https://example.com

curl's verification is strict and it has no AIA fetching, which makes it a good proxy for how your API clients will behave. If curl is happy on a clean machine, most things will be.

For reading the certificates themselves, everything from the SAN list to the extensions and the days remaining is in the certificate decoder, which parses the PEM in your own tab rather than uploading it. Private key blocks are refused rather than parsed, which is the behaviour you want from anything you paste certificate material into.

The fixes, and the one that is not one

• Missing intermediate: point the server at the full chain file and reload. Verify with s_client, not with a browser.
• Wrong order: rebuild the file leaf-first. Two minutes, and it removes a whole class of client-specific failure.
• Client trust store too old: update the runtime, or install the root explicitly. In Alpine containers the answer is usually that the ca-certificates package is not installed at all; in Python it is an outdated certifi; in Java it is cacerts from a JDK nobody has patched.
• Corporate TLS interception: add the proxy's root to the trust store of the runtime that needs it. Java and Node each have their own store and both need telling.
• Hostname mismatch: check the SAN list, not the Common Name. Chrome has ignored CN entirely since version 58, so a certificate with the right CN and a SAN list missing the hostname is rejected while looking correct.
• Expired anything: renew, then set up monitoring that checks the full chain rather than the leaf.

And the non-fix: curl -k, verify=False, rejectUnauthorized: false, NODE_TLS_REJECT_UNAUTHORIZED=0. Each of these turns off the check that was working correctly and leaves the connection open to anyone able to intercept it. They are debugging tools, not configuration, and they have an unpleasant habit of surviving into production inside a Docker image nobody rebuilt. If a certificate cannot be validated, fix the chain or add the root you actually trust.

One more thing worth doing after the chain is right: enable HSTS, so browsers refuse a downgrade rather than offering the user a button that bypasses everything you just fixed. The header and its max-age, includeSubDomains and preload semantics are in our HTTP header reference, along with the warning that preload is much harder to undo than to turn on.

Chain and trust store questions