DevOps

What Is HashiCorp Vault & How Does Vault Work?

Imagine you run an app that needs to log into a database. Maybe it logs in with a username and secret password. What's the best place to store these login credentials? It might store them in a file or include them in the code.

Furthermore, what if you have ten apps that follow the same practice? Sensitive data would get scattered in many places, making it hard to track and increasing the risk of exposing it to a malicious actor. This, among other problems, is why we need HashiCorp Vault: It makes securely managing secrets easier.

This article explores what HashiCorp Vault is, how it works, its features, and its use cases.

What is HashiCorp Vault

HashiCorp Vault is a powerful tool for securely managing and storing secrets, such as API keys, passwords, and other sensitive information. It provides a centralized location for managing access to these secrets and offers features like encryption, access control, and audit logging to help protect your secrets. With Vault, you can easily manage secrets across multiple environments, including cloud providers, on-premises data centers, and hybrid environments.

HashiCorp Vault comes in three flavors.

The free and open-source tool: The free tool is self-managed. That means you have to install, configure, and manage everything yourself.
HashiCorp Cloud Platform Vault (HCP Vault): The HCP Vault solution is a managed system; HashiCorp takes care of all backend administration tasks, allowing you to use it more easily.
HashiCorp Vault Enterprise: The Enterprise solution has extra features that are unavailable in the other two flavors. For example, you'll see a feature for Disaster Recovery Replication. In general, these are features geared toward large companies with large infrastructures.

HashiCorp Vault Features

Below are some of the features that make HashiCorp Vault such an effective tool.

Centralization of Secrets: Vault provides a central point from which you can easily manage and update Secrets across multiple environments, ensuring they are always up-to-date and secure. It also ensures that access to them is tightly controlled and audited, reducing the risk of unauthorized access or data breaches. It is an important step towards building a secure and reliable software system.
Encryption: Vault provides Encryption as a Service (EaaS) to enable security teams to fortify data during transit and at rest. So, even if an intrusion occurs, your data is encrypted, and the attacker would never get the raw data.
Authorization: Vault supports multiple auth methods, including GitHub, LDAP, AppRole, and more. Each auth method has a specific use case. Before a client can interact with Vault, it must authenticate against an auth method. Upon authentication, a token is generated.

Now we can understand why HashiCorp Vault is called an identity-based secret and encryption management system: it can store many types of secrets.

How HashiCorp Vault Works

The main objective of this tool is to control access to sensitive credentials. It does this by encrypting and storing them in a central location called a Vault. The Vault can be accessed through an API or a user interface, and access can be restricted using policies and authentication methods.

With Vault, you can manage secrets across multiple cloud providers and on-premise infrastructure. Additionally, it keeps a detailed audit log of Vault access activities.

HashiCorp Vault Use Cases

Below are practical use cases of HashiCorp Vault:

Key and Secrets Management

Secrets can be of two types:

Static secrets - They are long-lived and do not change often. For instance, a password used to access a database; we can store that in Vault. Such passwords will sometimes remain unchanged for a long time.
Dynamic secrets - These are unique to a client and generated on demand. Say an application needs to access a database. Vault gives it a temporary secret - that expires after a defined period - to access the database.

By associating dynamic secrets with a lease, Vault ensures that the credentials are automatically destroyed once the lease expires. Additionally, Vault offers support for dynamic secrets across various systems and can be conveniently extended with plugins.

One advantage of dynamic secrets is that they only come into existence when they are accessed, thereby eliminating the risk of theft or duplication by other clients. Moreover, Vault has revocation mechanisms in place to promptly revoke dynamic secrets after use, which reduces the time the secret remains active.

If a hacker steals a static secret, they gain unlimited access to a database until the secret is changed. But if they steal a dynamic secret, they lose access after the lease expires.

Data Encryption

Organizations hold two types of data: data at rest and data in transit. Data at rest refers to data stored or saved in a particular location, such as a hard drive or database, that is not actively being accessed or transferred. Data in transit refers to data actively being sent or transferred between devices or networks.

Vault encrypts data at rest and in transit to protect against unauthorized access.

Encryption for data at rest - When secrets are stored in the Vault, they get encrypted. This adds a bit of protection to this data. If some unauthorized app or a hacker steals this sensitive data, they cannot read it.
Transit Secrets Engine - Vault uses a transit secrets engine to enhance the security of your data during transmission. This service allows security teams to encrypt data, which adds layer of protection against intrusions. To use it, an app requests Vault to encrypt the data before transmission. See the graphical representation below:

Identity-Based Access

Vault's authentication process involves verifying user or machine-supplied information against internal or external systems. Various auth methods, such as GitHub, LDAP, AppRole, etc., are supported by Vault, each catering to a specific use case.

Let's imagine a trip to the bank. What happens when we want to change some details of our bank account? We need to prove our identity to the bank's workers. We need to prove that we are authorized to request these changes. And we can do that by providing a national identity card. This proves, for instance, that someone is John Smith, the account owner.

A similar thing happens with Vault. When an app requests something, it has to prove its identity. This way, Vault can ensure it hands out secret data to the correct app. This prevents rogue apps from stealing data that is not meant for them.

Case Studies of HashiCorp Vault

Below are some of HashiCorp Vault case studies:

Adopting Vault for Growing Infrastructure (Yahoo!)

Yahoo! is one of the largest informational websites in Japan. Around 80 million users access its services. To cope with the growing demand, Yahoo! had to shift to a cloud-native architecture. And they had to use more and more applications.

These needed access to keys such as database credentials, API keys, and so on. Thus, it was essential to issue, manage, and securely store secrets without leaking them. It became necessary to implement a key management system. And it had to be able to securely manage credentials over multiple data centers without any performance issues.

The company initially thought of developing an in-house key management system. However, developing such a tool from the ground up would have been very time-consuming.

The company started looking for multiple third-party applications that would fit their requirements. The solution they wanted also had to be compatible with HSMs (Hardware Security Modules). These protect encryption keys using physical devices.

Finally, the company decided HashiCorp Vault is the perfect solution. This gave them several advantages:

Using an already existing tool was much faster than developing their own.
They could create a highly available key and secret management system. Highly available means it rarely breaks or goes offline.
Vault also supported its Hardware Security Modules.

In a nutshell, this tool fulfilled all of Yahoo's needs. It could reliably process the huge number of requests its infrastructure sends to a key/secret management system. More details on this case study can be found here.

Upgrading Multiple Vault Clusters (Wix)

Wix is a cloud-based web development service. And it has to manage multiple Vault instances. A collection of many Vault instances is called a cluster. When a new version of Vault is released, it becomes a tedious task to upgrade tens or hundreds of instances manually. So, Wix had to come up with a solution.

They created a CD (Continuous Deployment) pipeline that automatically upgrades all Vault clusters. But what's so interesting about this? Well, think about what usually happens when you upgrade something. You have to temporarily pause that service while you upgrade. And you also have to restart it after it's done. This could mean Wix users not being able to use the website for hours. They didn't want to upset users this way. So, they developed an automatic upgrade method that doesn't interrupt the website's services. Users won't even know clusters are upgraded in the background! And this is possible because Vault can be deployed in highly-available setups. That's one of its most powerful features.

You can think of it this way. Imagine you have one Vault server. If this breaks down, you lose all access to secret management services. But what if you have three Vault servers? Well, if one breaks down, two of them are still functional. So, key and secret management services will still work. Vault has the ability to have multiple instances work in parallel. This helps it achieve that high availability.

Read this if you want the full details about how Wix implemented their CD pipeline.

Growth of HashiCorp Vault

Since its inception in 2015, HashiCorp Vault has grown at a tremendous pace, becoming one of the most popular secret management tools. According to CNCF's 2021 End User Technology Radar Report, Vault has the broadest adoption across multiple companies in different industries.

As more and more companies adopt this tool, they will need people to manage these systems. The demand for people who can use HashiCorp Vault will only increase.

So start learning today! KodeKloud has a great, easy-to-follow HashiCorp Certified: Vault Associate Certification course.