It's no secret that Kubernetes Secrets are just base64-encoded strings stored in etcd alongside the rest of the cluster's state. Ever since the introduction of Secrets in 2015, armchair security experts have been scoffing at this decision and seeking alternatives. I think those people are missing the point.
The design of the Secrets API dates back to before Kubernetes v0.12. In a thread the predates the original design document, there's a line that hints at why people might be confused by Secrets:
Its hard to evaluate these alternatives without a threat model
That's exactly the issue. The naive approach to securing software is to blindly implement a checklist of security features. But a deeper understanding of security will quickly uncover that perfect security is impossible; you have to make trade-offs and prioritize the most likely scenarios. Creating a threat model can help you make those decisions. Let's create a rudimentary threat model for Kubernetes Secrets and see what comes up.
A simple threat model for Kubernetes Secrets
What are we protecting?
Secrets are generally used to store things like database passwords and private keys meaning they are a high-value target.
What does a security failure look like?
If an attacker is able to read a secret, they can use it to perform further attacks such as stealing data, modifying/deleting/ransoming data, or gaining authorization to do things like spawn pods that mine crypto. Normally we'd use something like DREAD to rank the severity of different attacks, but exposing secrets is somewhat binary unless we have a specific secret in mind.
How can secrets be stolen (what can go wrong)?
At a bare minimum, secrets need to exist in plaintext in the memory of whatever application needs it, where it can (almost) always be stolen by another process on the same node with enough perseverance. We also need to store the secret somewhere persistent. In our case, secrets are stored inside etcd and accessible from the Kubernetes API.
Since the secret must exist in those two places, they can be stolen in any of the following ways:
- A malicious process on the same node (scan memory, or if not enforcing securityContexts, read straight from /proc or the CRI)
- Root access to a control plane node (read etcd's memory, read the on-disk dump, or steal a client cert and connect directly)
- Root access to a worker node (steal kubelet's client cert and read the secret from the API server, or read the secret file/env var directly)
- Access to the physical server of a control plane node (plug the hard drive into another computer and read the etcd data or just dump the RAM)
- Future unexpected attacks (this is a catch-all which helps us pick solutions that have a smaller attack surface)
Some of the more eccentric hacks like social engineering, malicious insider, human error/misconfiguration, or hardware supply chain attacks are of course possible, but outside the scope of what Kubernetes can fix realistically.
How can we prevent those attacks?
For Attack #1: Stealing secrets from memory is a risk we're forced to tolerate. Applications could use auto-expiring tokens or multi-factor authentication, but those features are out of scope since they are application-dependent.
For Attack #2 and #3: Root access to nodes is a huge concern. This can be mitigated by general server hardening, patching, and preventing privileged pods from running, but this is a very complex threat to address.
For Attack #4: Access to physical server can be somewhat be mitigated by encrypting disks at rest. Crucially, the encryption key MUST be stored in a separate security domain to get any security benefit. But since physical access is typically game-over, you just need some serious physical security.
For Attack #5: Here is where we have to gamble on future zero-days appearing. We can raise our chances by choosing simpler and well-tested methods, and it doesn't get much simpler than plain Kubernetes Secrets.
Takeaways from the threat model
The threat model exposes an inconvenient truth that storing secrets is hard since the plaintext version has to exist somewhere (in contrast to e.g. password hashes). That's just the problem with reversible encryption.
Any improvement on the existing Secrets implementation would have to mitigate more of those attacks, but I posit that none of the proposed alternatives to plain Kubernetes Secrets offer enough extra security to be worth the hassle.
Alternatives to Kubernetes Secrets
Let's looks at some of the alternatives that exist and see how they measure up.
Etcd encryption at rest
I'm shocked this is still the #1 recommended alternative considering how wildly useless it is.
Etcd encryption at rest involves encrypting all Secrets inside etcd with a key that is... on the same filesystem as etcd itself. So none of the four attacks in our threat model are mitigated here. Not even the "physical access" attack since the key is stored on the same disk! Or at least another disk that is accessible from the same host (not even an option mentioned in the docs).
Etcd encryption via KMS
You can replace your encryption key from the above method with a Key Management Service from your favorite cloud provider.
While this is listed as the "Strongest" method, it's basically just as insecure according to our threat model. An attacker who can access the node can just mimic what etcd does and decrypt the secrets before exfiltrating them. At least this mitigates physical access to the disk, if and only if the KMS client is authenticating to your cloud provider with an auto-rotating, multi-factor token.
Using this option requires a hard dependency on your cloud provider, a lot of complexity, and a BIG blast radius if it ever breaks. If you're forced to encrypt secrets at rest for compliance, this is unfortunately your best option despite it not tangibly benefitting your security posture.
Bitnami Sealed Secrets
Sealed Secrets really aren't an alternative to secrets, but I've seen people think they are. Sealed Secrets allow you to store encrypted secrets in version control. When you kubectl apply a SealedSecret to your cluster, it gets automatically unencrypted and converted into plain Kubernetes Secrets by the Sealed Secrets controller.
Since SealedSecrets turn into plain Secrets, no attacks in our threat model are mitigated. If you have no other safe place to store Secrets, SealedSecrets is a good option, but our threat model considers out-of-cluster storage of secrets to be out of scope.
Vault Sidecar Injector
Here's the big one people point to. At its core, Vault is just a key-value store with a few key features:
- A clever Shamir sealing process, which people immediately disable in favor of auto-unsealing which negates the benefits of sealing just like etcd encryption via KMS.
- A rich policy language, which few people bother to learn.
- Great auditing, which no one monitors.
So in the end, Vault is just a key-value store unless you're paying $$$ for a managed Vault instance or a team of in-house Vault experts. I've worked for a company where we had a whole team running HSM-backed Enterprise Vault, but that thing still went down all the time.
But let's say you have the deep pockets for an impossibly-well-maintained Vault instance. You've just installed the Vault Sidecar Injector in your Kubernetes cluster. Do you get enough security out of this complex arrangement to be worth it? I'd argue no.
The sidecar injector works by modifying pods to have a Vault client sidecar which authenticates to your Vault server, downloads the secret, and stores it in a shared-memory volume which your app can access like a regular file.
For Attack #1: Since the secret is still in memory, attackers on the node can still steal it.
For Attack #2 and #3: If an attacker hacks any node (worker or control plane), they can run any pod with the right Vault annotations and steal the secrets.
For Attack #4: If someone accesses the physical node, they can't get the secrets from disk, but they can get vault credentials (tied to the serviceaccount with a plain Secret) and steal the secrets that way if you are running Vault inside Kubernetes.
However, you still have to worry about physical access to the server where Vault is running. Vault encrypts data at rest when it is "sealed", but if you're using auto-unsealing, an attacker can mimic that process using your on-disk cloud credentials. Heck, someone with physical access to your server need not bother with reading your disk; they can just dump the RAM directly if you have a free PCI slot.
For Attack #5: The complexity of running Vault greatly increases your attack surface. I trust HashiCorp to catch issues more than most companies, but more moving parts is always more risk. Sometimes that risk is worth it (like yes, hashing passwords is more complex than not, but the pros clearly outweigh the cons), but only if some of the other 4 attacks are mitigated.
So according to our threat model, using Vault introduces a few layers of indirection, but ultimately does not address more attacks than plain Kubernetes Secrets. Just using encrypted disks and storing the key somewhere safe would provide the same level of security MUCH more simply and cheaply.
Conclusion
By creating a threat model that includes the kinds of attacks you want to mitigate, it's clear that managing secrets safely is extremely difficult. The problem is NOT that secrets are just base64 encoded; that was never meant as a security feature. And the problem cannot be simply waved away by software/cloud providers and their flashy documentation.
For something as security-sensitive and difficult as storing secrets, start with a threat model. If multiple solutions have similar security according to the threat model, pick the simpler one to reduce the overall attack surface.
Edited to improve cohesiveness 2022-05-01