New Report: Most common errors when implementing HPKP, HSTS and preload conditions

We have collected and visited two different sources of domains and webpages, Alexa top million domains, and Shodan. These results come from November 2016 searches. From those domains, we have restricted the search to be able to determine which ones use HSTS or HPKP over HTTP or HTTPS, and even which of them uses different configurations for the headers. We have tried to determine not only the quantity but the «quality» of the implementation. Just 0,02% of most popular domains are implementing HPKP in the best possible way, and just 0,74% are doing so with HSTS. Even Whatsapp.com or Facebook.com have some errors.

We show now some excerpts from the report you cand find here.

Number of pins

When implementing HPKP it is important to respect the number of pins required. Despite the recommended values are using between 3 and 4 pins, some domains use from just one pin (violating the RFC) up to 17, which seems to be an irregularity that reduces the efficiency. Regarding Alexa top million domains, 282 out of 450 domains use 2 or 3 pins, which is correct. 89 (19,8%) use zero or just one, which is useless from the browser standpoint since it will ignore it.

Number of pins offered by top 1 million Alexa domains using HPKP.

Which certificate to pin

When using HPKP, choosing the right certificate to pin may be an important decision. Administrators may use whatever pin in the chain (root, intermediate or leaf) but this decision may impact directly in their usability and security from the administrator standpoint and user security. There is a tradeoff between security and maintenance.

• Pinning the root offers less security, but an easier way for the administrator to deal with HPKP. This means that, as long as the administrator does not change its CA provider, no additional changes should be done, so less maintenance is required. But, on the other hand, if an attacker gets a fake certificate from the same CA, the browser would not detect the difference, since the root remains the same.
• Pinning the intermediate certificate is the best choice, maybe. The attacker should get a certificate from the same subCA to make the «perfect» attack. The administrator, on the other hand, may change its leaf certificate as long as it comes from the same subCA with no extra cost of changing pins.
• Pinning the leaf is the most secure way, but the most «dangerous» as well. If the certificate expires or for whatever reason the certificate changes (more specifically, the public key), even if issued by the same CA or subCA, the administrator has to modify its pins or use the backup one. On the other hand, an attacker may not be able to create a valid certificate (unless the private key is stolen) to create a man in the middle «perfect» scenario.

So we have checked what certificate does administrators pin, and this is what we have found. Most of them (73,65%) use the intermediate certificate to pin.

Pinned certificates in the trust chain for the top million Alexa domains using HPKP.

Pins reuse

Reusing pins among different domains is not an invalid practice at all. Considering that most of the pins used in HPKP are «intermediate» pins mostly from subCAs, it is even absolutely normal to share some pins between domains. But this procedure brings a little risk. Thus, from an attacker standpoint, knowing which subCAs or even CAs are pinned may allow to plan a specific APT for that domain. For example, if a domain issues its intermediate certificates with a specific subCA and pins this intermediate certificate, an attacker that gets a rogue leaf certificate for that domain issued from the same subCA will still have a perfect MiTM situation, since the browser will not show any warning message. Therefore, from the attackers standpoint, if they are able to determine if a domain pins its intermediate certificate, and furthermore, which one is the pinned subCA, it allows him to know better who to target. Additionally, if the attacker wants to maximize its scope, he would try to get a rogue certificate signed by this «popular» subCA.

The following map represents which certificates (and its pins) are pinned with more domains. These are the top 25 most pinned certificates. Since the protocol allows to know just the pin and not the certificate itself, it is necessary to «unhash» the certificate. We have collected several millions of certificates and hashed them to compare it with the pins associated to the domains. The results show how an intermediate certificate from Comodo is the most pinned certificate (klO23nT2ehFDXCfx3eHTDRESMz3asj1muO+4aIdjiuY=). It pins 40 different domains from Alexa and Shodan.

Pins reuse map. Click to enlarge.

Preload

To avoid «Trust on first use» issue, «preload» mechanism was introduced. This preload works as a root CA embedded in the browser. It is basically a list of domains that are willing to be accessed with HSTS securely from the first time. This list is maintained by Google and some conditions have to be satisfied to belong to this list.

• Have a valid certificate chain and redirect from HTTP to HTTPS in the same host (of course)
• Serve all subdomains under HTTPS. WWW is mandatory if it exists in DNS server.
• Serve HSTS header via HTTPS with this properties:
• max-age is at least 18 weeks (10886400 seconds).
• includeSubDomains directive must be included.
• preload directive must be included.
• If serving an additional redirect from the HTTPS site, it must still use the HSTS header (rather than the page it redirects to).

If all these conditions are satisfied, the domain owner may apply to the list in here: hststpreload.appspot.com and the domain will be eventually included in the list. This webpage allows as well to check if a domain satisfies or not all these conditions. There are a total of 18197 domains preloaded in Chromium list (shared with Firefox). As of December 2016, only 2056 domains from the top 1 million from Alexa are in that list.

Preloading status in Alexa’s top million domains

In the background, hststpreload.appspot.com uses a public API providing the reasons why a specific domain may be preloaded or not. We have checked all the top million Alexa domains against this API, to know if preloaded domains do really validate all this conditions to be preloaded. When a domain is checked against this API or preload list, the domain is visited in real time and errors checked. It is interesting to prove that, from those 2056 preloaded domains in top Alexa list, 662 contain some errors, thus, strictly speaking, they should not be preloaded. We have even detected that, 67 out of those 2056 preloaded domains in the list, do not contain the preload directive in the header, which as well violates the condition. Whatsapp.com and Facebook.com are domains that do not keep the mandatory conditions to be preloaded, but they actually are.

Conclusions

Although HSTS and HPKP protocols are intended to provide an additional layer of security to HTTPS communications, their implementation is not widespread. At server level, many of the most relevant Internet domains do not even implement them. Moreover, among the minority of domains that do use them, there exist a significant number of implementation errors, even a disregard of the recommendations of their respective RFCs. This situation shows both low level adoption and, somehow, some misunderstanding about how to take full advantage of these protocols. Some of the most interesting figures are:

• From Alexa, we have collected 632648 HTTPS domains, and 901958 HTTP domains. We retrieved 30886979 HTTPS (port 443) domains and 45330802 HTTP (port 80) domains (a total of 76217781) from Shodan.
• Only 1,9% of domains in Shodan use HSTS correctly over HTTPS, while just a 5,35% from the Alexa top million do so.
• 4717 (roughly a 0.74%) of the top million domains in Alexa using HTTPS (632648) are implementing HSTS in the best possible way.
• 175 of the top million domains in Alexa (a roughly 0,02%) using HTTPS (632648) are implementing HPKP the best possible way.
• 20% of top Alexa domains using HPKP over HTTPS use zero or just one pin, which is useless from the browser standpoint since it will ignore it. Most of them (a 73,65%) use the intermediate certificate to pin.
• 17% of domains in Alexa implementing HPKP are using a wrong or ignored max-age value.
• The most used pin (a certificate from Comodo) pins 40 different domains from Alexa and Shodan.
• There are a total of 18197 domains preloaded in Chromium list (shared with Firefox). As of December 2016, only 2056 domains from the top 1 million from Alexa are in that list.
• From those 2056 preloaded domains in top Alexa list, 662 contain some errors if checked against the official preloading API, so, strictly speaking, they should not be preloaded. Whatsapp and Facebook are among those domains that do not keep the mandatory conditions to be preloaded, but they actually are.

Here is the whole report.