Lead Image © phive2015, 123rf.com

Lead Image © phive2015, 123rf.com

Secrets and certificate management

Locked Vault

Article from ADMIN 58/2020
Vault is a highly secure, trusted place to keep your secrets and certificates.

During a recent project, I needed to test against an implementation of HashiCorp's Vault [1]. If you haven't come across Vault before, it's the industry leader for cloud-native secrets and certificate management that, by design, also plays very nicely with software in the cloud, such as Kubernetes. According to the Vault website, you can safely keep and fine-tune access for users and applications for "tokens, passwords, certificates, encryption keys … and other sensitive data using a UI, CLI, or HTTP API."

Working on security in DevOps teams, I had seen and used Vault, built by Terraform, on two or three different Amazon Web Services (AWS) estates but had never set up a lab myself on a local machine for testing. I wanted a simple solution that was slick and quickly reproducible, and, because it was for a lab environment, it didn't have to support high availability (HA). Thankfully, the clever Vault provides what is called dev server mode for just that requirement.

In this article, I deploy the venerable Vault locally to offer a centralized, flexible, and highly secure place to keep secrets and certificates; then, I'll run through a few of its core features to get to grips with the basics.

Is It Safe?

I'll dive straight into getting Vault up and running. I'm using Linux Mint 19, which sits atop Ubuntu 18.04. The set up should be mostly the same configuration, however, if you use another Linux distribution.

First, I'll navigate to the binary download page [2] and choose Linux 64-bit . I'm going to become the root user in this terminal and in a second terminal a little later on. After clicking the Download button, a ZIP file starts downloading. In the following command, you can see the version I'm using as I decompress the file:

$ unzip vault_1.3.2_linux_amd64.zip

The next thing to do is move the resulting vault binary file to the user path:

$ mv vault /usr/local/bin

To see if it works, I enter:

$ vault --version
Vault v1.3.2

One nice touch is that the vault command can use autocompletion, just like the Bash shell. To switch that feature on, enter,

$ vault -autocomplete-install

which will drop the line complete -C /usr/local/bin/vault vault into your .bashrc file. Close and reopen your terminal to load that up, or in your home directory run

$ . .bashrc

to load that file into your shell again. Try hitting the Tab key with the vault prompt on your command line now, and you'll see a number of options, which is a great way of checking syntax when you're learning Vault.

Under Lock and Key

As mentioned, I'm going to create a dev server for the purposes of running a lab environment. The Vault docs warn you that it shouldn't be used in production, isn't very secure, stores loads of things in memory, and automatically initializes (or "unseals") the strong locking or "sealing" mechanism at install time. You will also lose your secrets when you kill the terminal in which the dev server launches. Consider yourself suitably warned! For more information, read the "Concepts" box.


If you are not familiar with Vault, I'd definitely recommend reading the Concepts docs page [3], especially for sealing and unsealing the Vault state. When using just a dev server, it's not a big issue where you save your temporary unseal key and root token, but it's critical that good practices are followed in any environment other than this temporary lab setup.

I'd encourage you to read the Seal/Unseal page, along with the other core concepts very carefully to avoid risking the loss of valuable certificates and secrets at some stage in the future.

All commands that you send from the vault binary (think of it as a client) are encrypted by TLS as they travel over to the Vault server. To fire the dev server up, use the command:

$ vault server -dev

The uppermost part of the abbreviated output in Listing 1 shows what you need to know. Note that you will need to open another terminal to continue with other commands from this point on, because the server will keep that window open as it runs in the foreground.

Listing 1

Starting Up the Dev Server

==> Vault server configuration:
        Api Address:
                Cgo: disabled
    Cluster Address:
         Listener 1: tcp (addr: "", cluster address: "", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
          Log Level: info
              Mlock: supported: true, enabled: false
      Recovery Mode: false
            Storage: inmem
            Version: Vault v1.3.2
WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.
You may need to set the following environment variable:
  $ export VAULT_ADDR=''
The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.
Unseal Key: PrAbjZWj2NYYnV6Lh/FRpf5Nu5f2E5fwZZf95JQiLdo=
Root Token: s.pKGs7KObPOp0Cx2ybiUWuEIW
Development mode should NOT be used in production installations!

After opening up a second terminal, you can now interact with the Vault server. For good measure, I'll offer the new terminal the VAULT_ADDR environment variable with the command

$ export VAULT_ADDR=""

so that the shell can access the correct IP address and TCP port for the server.

The next step is using an environment variable for the root token, copied from the dev server start-up output in the first terminal, within the second terminal:


From the second terminal, you can check that the client – the vault binary that provides its command-line interface (CLI)) – can connect to the server (Listing 2). The output shows a successful response, which means that your client to server comms are working as hoped.

Listing 2

Check Vault Status

$ vault status
Key            Value
---            -----
Seal Type      shamir
Initialized    true
Sealed         false
Total Shares   1
Threshold      1
Version        1.3.2
Cluster Name   vault-cluster-e5878f26
Cluster ID     54ef0ecc-5f31-0926-e904-1bd94bc152ae
HA Enabled     false

Lockdown, Lockdown

With a working Vault instance, I will now look at a few ways to make use of it. The first, most obvious thing to try is saving a secret to the secured vault. This secret could be any bit of data – not just a password:

$ vault kv put secret/chrisbinnie liverpool=best
Key              Value
---              -----
created_time     2020-03-13T12:56:24.857347664Z
deletion_time    n/a
destroyed        false
version          1

That output looks promising; Vault has responded as expected. If that doesn't work correctly for you for some reason, then you've probably not exported the two environment variables correctly, so check them. Take note that the secret/chrisbinnie part in the command above relates to the path where the secret is stored. The "Secrets Engines" box has more information to help explain how the path is configured. As you might expect, the liverpool=best element is the key-value pair.

Secrets Engines

A quick word about the path used when writing and retrieving secrets: In the example, the path element in the command was secret/chrisbinnie. The path tells Vault where to store a secret and within which secrets engine. Vault usually uses the version 2 key-value secrets engine as its default [4]. HashiCorp has more information on getting started with secrets engines on their website [5].

Think of a secrets engine in Vault as a type of virtual filesystem. The powerful flexibility that it offers means that Vault can interact with databases, hardware security modules (HSMs), and physical machines, to name but a few. As a result, AWS Identity and Access Management (IAM) and other authentication services can be integrated with relative ease. Note also that I'm using a RAM disk of sorts for my dev server; Vault is well designed and incredibly flexible.

Now that you know how to write a secret, I'll try to retrieve one with a get command (Listing 3). You can verify that you are seeing the correct secret by the timestamp in the command's output; also, you can see the key-value pair data under the Data section.

Listing 3

Retrieving a Secret

$ vault kv get secret/chrisbinnie
====== Metadata ======
Key            Value
---            -----
created_time   2020-03-13T12:56:24.857347664Z
deletion_time  n/a
destroyed      false
version        1
====== Data ======
Key         Value
---         -----
liverpool   best

Simply replace the get in the command in Listing 3 with delete to remove a secret. Note that it's possible to write multiple secrets to the same path. At an uber-basic level, that's how secrets are written, retrieved, and deleted in Vault.

However, another very powerful feature concerns dynamic secrets [6]. According to the docs: "Dynamic secrets do not exist until they are read, so there is no risk of someone stealing them … dynamic secrets can be revoked immediately after use, minimizing the amount of time the secret existed."

This sophisticated functionality is perfect for an environment such as AWS, wherein an IAM role is given access via a role with a policy attached to it. Vault then temporarily offers access to permitted resources with the use of dynamic credentials that can be revoked or set to expire within customized time periods. Additionally, ephemeral, single-use tokens are powerful.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Credential management with HashiCorp Vault
    Admin teams can use secret sharing to centrally manage shared access to user accounts and services. HashiCorp Vault is one of the few tools that has proven effective when it comes to implementing this solution. Here's how to use this open source tool and keep important credentials safe.
  • Automatic data encryption and decryption with Clevis and Tang
    Encrypting hard disk partitions during the installation of an operating system is standard procedure. When booting the computer, you then need to enter a matching passphrase to unlock the hard drive. We show you how to automate this process and link it to a policy.
  • Jenkins Configuration as Code
    The move from Groovy scripts to Jenkins Configuration as Code simplifies the initialization of Jenkins and Jenkins plugins.
comments powered by Disqus