Something that tends to happen as you start building things is that you end up starting in ‘get it running fast’ mode so that you can see how something works, knowing full well due to the in-your-face reminders and, well, reasons, that you’re not going to produce something that is ‘production ready’. You toy around with it, maybe you even spend a lot of time learning how to use it, perhaps you even do use it in some environment whether that might be at home, at work, for clients, or whatever and then you never seem to find the time or desire to circle back around and tighten it up. I can’t count how many times I’ve been down this road over the years!
While there’s nothing wrong with this approach for learning purposes, when it comes time to exercise your knowledge in an actual production environment with specific requirements that you can’t afford to just kick down the road, you’ve got that one thing you’ve never done: you haven’t actually deployed a production ready <insert whatever thing>.
These days probably the most common warning you’ll run across has to do with how you store secrets. I’m talking about passwords, API keys, a credential or some other data that you shouldn’t be storing in code. Taking that a step further, you shouldn’t even be storing those as local environment variables. The name of the game is stashing these things in some reasonably secure manner. Taking it another step further, and totally in scope of this post even, is understanding the importance of not hard-coding or storing environment-level configuration data. Secrets engines are perfect for centralizing and controlling configuration data that isn’t sensitive at all.
Probably the second most common trap is that most of the time you end up deploying a service without using SSL. Again, in your lab or test environments it may not matter per se, but you’re going to have to address SSL in live production environments and your lab / lower environments are not only the perfect place to implement SSL as a proof of concept, but it also builds good habits.
To be clear, this series is not a walkthrough of a fully-functioning production grade implementation of secrets, configuration, and SSL certificate management. What it is though, is a demonstration of building a secrets/certificate management engine suitable for extending your understanding and learning. You can then experiment in your own environment and implement a production-ready solution.
Hashicorp Vault is just one way to manage secrets and this post covers a simple deployment of Vault. Some products have their own secrets management subsystem that are quite good and perfectly suitable for many types of environments, including production ones. You have to weigh the pros, the cons, and the complexities of any implementation.
Things you need
Pretty simple… a host running Docker Engine. I use a bare-metal Linux machine at home for most of my blogging deployments.
The plan
- Bring up a basic instance of Hashicorp Vault without securing endpoints with SSL
- Turn Vault in to a Certificate Authority and issue yourself SSL certificate to secure Vault endpoints and whatever else interests you
- Re-configure Vault with SSL-secured endpoints
- Enable a secrets engine to store passwords, credentials, environment variables
Deploy Hashicorp Vault
With this simple Vault experiment, we’ll have a working Vault we can store things in. We can then see what exactly it means to not store things in code and actually how to not store things in code.
You will need a working Docker runtime. I use a bare-metal Linux machine in this case and simply add the Vault container to my compose stack.
In this case, I’ve got a /data1 mount on my machine and I’m using the ‘file’ Vault backend, which is exactly that… things get stored on the local filesystem. I create the directories /data1/vault/{logs,file,config} and then create a simple Vault config in the config directory:
cd /data1
mkdir -p vault/{logs,file,config}
cat > ./vault/config/vault.json << EOF
{
"backend": {
"file": {
"path": "/vault/file"
}
},
"listener": {
"tcp":{
"address": "0.0.0.0:8200",
"tls_disable": 1
}
},
"ui": true
}
EOF
Now, the relevant Docker compose (${DATADIR} is set in my .env as /data1)
vault:
image: vault:latest
container_name: vault
restart: always
ports:
- "8200:8200"
cap_add:
- IPC_LOCK
volumes:
- ${DATADIR}/vault/logs:/vault/logs
- ${DATADIR}/vault/file:/vault/file
- ${DATADIR}/vault/config:/vault/config
entrypoint: vault server -config=/vault/config/vault.json
Now start the Vault container…
docker-compose up -d
Vault will initially come up uninitialized. Initializing Vault is a simple process that can be done either by browsing to your Vault instance at http://X.X.X.X:8200 or via vault cli as follows. You will need to install the vault cli for your operating system. E.g. mac users can use ‘brew install vault’ for instance. I like the Vault UI, but I’ll go with cli here…
Once you have vault cli, set an environment variable VAULT_ADDR pointing to your Vault instance:
export VAULT_ADDR='http://10.0.0.10:8200'
Now, initialize the vault. Be sure to save all of these keys safely in an appropriate password safe of some kind.
vault operator init -key-shares=6 -key-threshold=3
Unseal Key 1: kOksKgWWILFv...invVHwJ/2cwJUS
Unseal Key 2: Slo/F9ZupS/...inSMA0P64WMvRw
Unseal Key 3: F3iobOrrpLz+O...kWwteEP4/dxojkE
Unseal Key 4: EFLfT80AnEU...qU/eINF48Vvg
Unseal Key 5: IfMp7lh5zlpq6WhQ...5fxdOumz3
Unseal Key 6: 0CNal2hYOB...7JCKSiikrg60
Initial Root Token: s.44N4...ZmZ9T
Vault initialized with 6 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
As I said, I like the Vault UI, you could switch over to the vault at this point and provide 3 unseal keys to the UI, but here’s the cli version of it… when prompted just paste in one of your unseal keys.
vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 6
Threshold 3
Unseal Progress 1/3
Unseal Nonce 428c2bc9-9fdb-024e-383a-51843832eea9
Version 1.6.0
Storage Type file
HA Enabled false
Notice the output now says Unseal Progress 1/3 which lets us know that we’ve successfully provided 1 of 3 unseal keys. Sealed is still ‘true’ meaning the vault is still sealed. Repeat the process and enter another one of your keys. You’ll see similar output but now with an Unseal Progress of 2/3. Sealed will still be true.
On the third and final unseal key, the output will update to reflect Sealed ‘false’ and Unseal progress 3/3.
vault operator unseal
Unseal Key (will be hidden):
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 6
Threshold 3
Unseal Progress 1/3
Unseal Nonce 428c2bc9-9fdb-024e-383a-51843832eea9
Version 1.6.0
Storage Type file
HA Enabled false
The Vault is now unsealed!!
In the next part of this series, we’ll turn Vault in to our own Certificate Authority!
