Thoughts on Secrets Management
There are some great tools out there for secrets management, but with my current budget of zero these are a bit beyond my reach. Hashicorp’s vault solution does have an open source offering for an on-premise implementation and Doppler offers a free developer edition for their hosted service that might be worth investigating.
But first a bit of background information on secrets:
What is a secret
Secrets are objects that either through inspection or use provide the holder with additional privilege or access to a platform. Some examples include:
- Passwords.
- Asymetric key pair private keys.
- SSH private keys.
- PGP private keys.
- Certificate private keys.
- JWT tokens.
- Custom header keys providing API authentication or authorization.
Is Encrypting them enough?
In short, no. Security should be built in layers. Encryption is a great start, but if I host the credentials to a database in my public github repository, then I’m now completely reliant on the encryption algorithm I’m using being robust enough to handle people attempting to crack it.
As a general rule of thumb:
Data that is considered privileged or private information should only be accessible to those who need it – even in encrypted form.
##General Secret Guidelines Storage – secrets should be stored in a manner that makes it difficult for information to leak (encryption, physical security), but also ensures they are unlikely to be lost (backup, high availability or replication).
Access – only the people and systems that need a specific secret should be able to obtain it.
Replication – secrets, when obtained should be present only as long as required.
Audit – a journal should be kept detailing when any given secret is created, accessed and changed, and by whom.
The people element – the process in place should be sufficiently user friendly that people don’t start building their own processes to work around it. This is often the most significant area of risk.
My requirements I have secrets both inside and outside of the Kubernetes platform. The Kubernetes secrets manager can certainly provide what is required internal to the platform but this doesn’t cater for external objects that I need to access. Its also critical to note that Kubernetes doesn’t encrypt etcd data by default, so using this system will require some minor changes to the cluster.
A localy hosted Vault? There are several offerings for local secrets management such as Hashicorp’s Vault platform. My concern with using this solution is that I lack a dedicated host to store this on. In my current implementation I’d likely need to install Vault on my management pane alongside my logging and administration tools, and this poses some risk.
A vault by design should be as separated from the systems it is providing secrets for as possible without impacting usabilty. The reason for this is to reduce the potential for exploit or for accidental leakage of secure data.
In a general implementation, a software vault holds the keys and details to decrypt and retrieve its own data. That means that if a malicious actor obtained access to this system it is quite possible that a memory dump or a similar method could provide all the data I’ve been working to protect.
We reduce the risk of this by severly limiting access to the platform, so getting in for maintenance should be tricky, the system shouldn’t provide other services (I shouldn’t be logging in to use this host for any other reason), and we should ideally segment this system away with firewall rules and watch it very closely with logging, audit and vulnerability detection tools.
Alternatives?
Given my situation, the most reliable solution would be to pay for a hosted service. This takes most of the pain away from setting up a service, but comes with added costs that are trivial for an enterprise, but not so much for me.
Doppler offer a developer package that could suit the needs of this platform as an experimental solution which might be a reasonable fit for now.
Given any other implementation my strong recommendation would be to adopt a SaaS approach. Secrets are important, and if managing them isn’t part of your core business, you should probably partner with someone who does this for a living to make sure this is done well.
I could also roll my own solution, leveraging Sops for encryption/decryption of secrets, but my concern there is that I’m probably going to expose myself to more risk building out my own solution than in accepting the risks inherent with a Vault hosted alongside some other tools.
Revisiting my objective
So the purpose of this dev platform was for experimentation and learning. Given the three options I’ve listed as available within my budget:
- Implement a local Hashicorp Vault instance.
- Sign up for the free version of Doppler.
- Roll my own. I believe experimenting with the SaaS option in Doppler would be the most beneficial from a learning perspective, so I’m going to start there and document how I go.