What Is HashiCorp Vault and Why Should You Learn It?

Imagine you run an app that needs to log in to a database. Maybe it logs in with a username and secret password.

Username: app_user

Password: 3juf4foflw212s

Where can the application store these login credentials? It might store them in a file. It might even store these in its own lines of code. But it's insecure to store sensitive information this way. Furthermore, what if you have 10 apps that do this? Sensitive data gets scattered in many different places.

HashiCorp Vault tries to solve these problems.

1. First of all, it gathers all data in one place. It centralizes it. Now you have a single location for those secrets. It makes it easier to manage and find this sensitive data.
2. Next, HashiCorp Vault does its best to secure secrets. For example, it can encrypt sensitive data. So if someone manages to steal it from Vault, they shouldn't be able to read it. They won't see "password: 3juf4foflw212s". They will see some encoded data instead, which cannot be deciphered. So the password should still be safe.
3. Another important thing it does is authorization. Let's say an application requests that password. Is it the real app, or is it some fake app created by some hacker? And if it is the real app, should it be allowed to access the password? Vault tries to ensure sensitive data is only handed to the correct apps.

Now we can understand why HashiCorp Vault is called an "identity-based secret and encryption management system". There are many types of secrets it can store. Just a few examples are passwords, encryption keys, certificates, tokens, SSH keys, and so on.

HashiCorp Vault comes in three flavors.

1. The free and open-source tool which you can install on your infrastructure. And the premium (paid) solutions:
2. HashiCorp Cloud Platform Vault (HCP Vault).
3. And HashiCorp Vault Enterprise.

The free tool is self-managed. That means you have to install, configure and manage everything yourself. The HCP Vault solution is managed by teams at HashiCorp. They basically take care of some administrative tasks. So you can use this service more easily. Finally, the Enterprise solution brings in extra features, not available in the other two. For example, you'll see a feature for Disaster Recovery Replication. In general, these are features geared towards large companies with large infrastructures.

Use Cases of HashiCorp Vault

Key and Secrets Management

Secrets can be of two types:

1. Static secrets - They are long-lived and do not change often. For instance, a password used to access a database; we can store that in Vault. This password may remain unchanged for a long time.
2. Dynamic secrets - These secrets are short-lived and generated on demand. Say an application needs to access a database. Vault gives it a temporary secret, so it can write to that database. And Vault can also revoke this secret soon after the app is done. This way a secret changes more often. It can create a harder target for a hacker. Imagine if they steal a static secret. They might get access to a database for 1, 2, 3 months, until we change that secret. But if they steal a dynamic secret, they might lose access tomorrow, when that secret changes.

Data Encryption

Some data stays in the same place. This is called "data at rest". But other pieces of data move around. This is "data in transit". Vault can use encryption for both of these, to improve security. Let's see how.

1. Encryption for data at rest - When secrets are stored in the Vault, they get encrypted. This adds a bit of protection to this data. If some unauthorized app or hacker steals this sensitive data, they won't be able to decode it.
2. Encryption as a service (Transit Secrets Engine) - In this case, Vault encrypts data that needs to travel, from point A to point B. Suppose an application needs to send some sensitive information, like a phone number. This should not be transmitted without protecting it. To secure data in transit the application would need to encrypt it. But what if the application does not know how to encrypt? It can ask HashiCorp Vault for some help.

The "chat" between the app and Vault goes something like this:

App: Hey Vault, can you encrypt this phone number for me?

Vault: Sure thing! Here is your encrypted data.

App: Thanks!

The application can now safely send this encrypted data. Furthermore, Vault can also decrypt data on behalf of other apps. If some database app received that encrypted phone number, it can ask Vault to decrypt it.

Basically, this lets apps worry about their main job. They don't need to know how to encrypt or decrypt. Vault can do this for them.

Now, especially if you're a developer, you might wonder "How does encryption as a service help?" Well, you don't need to worry about implementing encryption/decryption in your own app. You can just offload this job to HashiCorp Vault. So you save a lot of time and write less code. Furthermore, encryption is hard to do correctly. HashiCorp tries to ensure its encryption mechanism doesn't have any security holes.

Identity-Based Access

Let's imagine a trip to the bank. What happens when we want to change some details of our bank account? We need to prove our identity to the bank's workers. We need to prove that we are authorized to request these changes. And we can do that by providing a national identity card. This proves someone is John Smith, the owner of that account.

A similar thing happens with Vault. When an app requests something, it has to prove its identity. This way, Vault can ensure it hands out secret data to the correct app. This prevents rogue apps from stealing data that is not meant for them.

It's useful to mention at this point that human beings can also interact with Vault. It has mechanisms to authenticate humans too, not just applications.

Case Studies of HashiCorp Vault

Adopting Vault for Growing Infrastructure (Yahoo!)

Yahoo! is one of the largest informational websites in Japan. Around 80 million users access its services. To cope with the growing demand Yahoo! had to shift to a cloud-native architecture. And they had to use more and more applications. These needed access to keys such as database credentials, API keys, and so on. Thus, it was essential to issue, manage, and securely store secrets, without leaking them. It became necessary to implement a key management system. And it had to be able to securely manage credentials over multiple data centers, without any performance issues.

The company initially thought to develop an in-house key management system. But it would have been very time-consuming to develop such a tool, from the ground up. And it would have been even harder to ensure it's very secure. That needs many months of work, real-world tests, and security audits. The company started looking for multiple third-party applications that would fit their requirements. The solution also had to be compatible with HSMs (Hardware Security Modules). These protect encryption keys using physical devices.

Finally, the company decided HashiCorp Vault is the perfect solution. This gave them several advantages:

• It was much faster to use an already existing tool, rather than develop their own.
• They could create a highly-available key and secret management system. Highly available simply means "more reliable." It rarely breaks or goes offline.
• This also supported their Hardware Security Modules.

In a nutshell, this tool fulfilled all of Yahoo's needs. And it could deal with the huge number of requests their infrastructure sends to a key/secret management system. More details on this case study can be found here.

Upgrading Multiple Vault Clusters (Wix)

Wix is a cloud-based web development service. And it has to manage multiple Vault instances. A collection of many Vault instances is called a cluster. When a new version of Vault is released, it becomes a tedious task to upgrade tens or hundreds of instances manually. So Wix had to come up with a solution.

They created a CD (Continuous Deployment) pipeline that automatically upgrades all Vault clusters. But what's so interesting about this? Well, think about what usually happens when you upgrade something. You have to temporarily pause that service while you upgrade. And you also have to restart it after it's done. This could mean Wix users not being able to use the website for hours. They didn't want to upset users this way. So they developed an automatic upgrade method that doesn't interrupt the website's services. Users won't even know clusters are upgraded in the background! And this is possible because Vault can be deployed in highly-available setups. That's one of its killer features.

You can think of it this way. Imagine you have one Vault server. If this breaks down, you lose all access to secret management services. But what if you have three Vault servers? Well, now if one breaks down, two of them are still functional. So key and secret management services will still work. Vault has the ability to have multiple instances work in parallel. This helps it achieve that high availability.

Read this if you want the full details about how Wix implemented their CD pipeline.

Growth of HashiCorp Vault

Since its inception in 2015, HashiCorp Vault has grown at a tremendous pace. It became one of the most popular secret management tools.

Check out this CNCF (Cloud Native Computing Foundation) End User Technology Radar Report published in January 2021. HashiCorp Vault has the broadest adoption across multiple companies in different industries. This confirms that Vault is one of the most used secret management systems.

As more and more companies adopt this tool, they will need people to manage these systems. The demand for people that know how to use HashiCorp Vault will go up. It's also useful that Vault integrates well with other tools, services, and companies. The integration program had a 300% growth since 2019. This means that companies are able to use Vault almost anywhere they want. They can integrate it with their preferred cloud solutions. They can interconnect with other services they want to use. Vault is flexible. This makes it easier for companies to adopt it. You can imagine the demand for skills related to HashiCorp Vault will grow for a long time.

So start learning today! KodeKloud has great, easy-to-follow courses on HashiCorp Vault and other HashiCorp tools, like Terraform.