Only Oteemo transforms business through acceleration, enablement, and adoption
Oteemo uniquely transforms teams and processes too
Why we’re different
Get to know us
Work with us
Cloud, Security, Kubernetes
by Chris McGrath | Jun 20, 2019
When I started working with Kubernetes and Infrastructure as Code, I quickly found out that I needed a secrets management solution, but when I googled around there didn’t seem to be a solid consensus on a best practice approach that could be universally applied to all situations. So, earlier this year I set a goal for myself to discover what application and infrastructure secret management solutions exist, come up with which one I thought was best and develop a working mastery of it. While pursuing this goal, I came to the conclusion that HashiCorp Vault is overhyped and Mozilla SOPS with KMS and Git is massively underrated.
I think SOPS is underrated for two main reasons:
Most of Vault’s hype is warranted as for decades there were no good secrets management solutions, and then here comes Vault from the makers of Terraform, with built-in secrets rotation, actively maintained over time, with great docs, support, and a community, and Vault was the only* solution that met my requirements for what the ideal secrets management solution looks like. I say Vault’s overhyped because I often see it recommended as the gold standard cure all that should be applied to all secrets management scenarios.
*Note: Mozilla SOPS also met my requirements, but I didn’t realize it at the time because I originally thought there was no safe way to do git encrypted secrets.
Whenever I found a secrets management solution I noticed I could group it into 4 main categories:
With my analysis complete, I spent a month of spare time working on a Vault Server for storing static secrets to help me gain a working mastery of Vault, I wanted it to be secure, easy to maintain, and easy to use. I did my best to achieve this by enabling TLS, adding Vault Configuration, Roles, Policies, and Kubernetes Infrastructure as Code for a highly available Vault/Consul Cluster to a git repo, using KMS auto unseal, writing good readme documentation, enabling versioned key-value store, LDAP authentication, the web GUI, and a third party desktop GUI called Cryptr by Adobe.
While learning Vault I noticed many drawbacks to its usage:
Given these drawbacks, I decided to dive deeper and research further, that research lead me to Soluto’s Kamus where I was introduced to 2 cool concepts: GitOps and zero-trust secrets encryption. That got me leaping through a rabbithole of encryption techniques. At the end of the journey I came up with the following mental schema.
1.) Symmetric Encryption Keys:
2.) Asymmetric Encryption Public-Private Key Pairs:
3.) HSMs (Hardware Security Modules):
4.) Cloud KMSs (Key Management Services):
My new understanding of advanced encryption techniques lead me to realize that KMS could be leveraged to prevent decryption keys from being leaked. That plus the ability to revoke decryption rights without needing to make any changes to encrypted files makes truly secure encrypted files in Git a reality. I revisited a few Git based encryption solutions I’d previously dismissed and discovered that Mozilla SOPS satisfied all of my criteria for an ideal secrets management solution. It also integrates well with CICD pipeline tools: There’s a SOPS Terraform Provider, Helm Secrets is just a wrapper for SOPS, and you can always fallback to:
Bash# sops --decrypt mysecret.yaml | kubectl apply -f -
(where kubectl could have been any CLI Utility that accepts standard input (-))
SOPS has none of the drawbacks of other Git based encryption solutions:
One of the footguns in other Git based encryption solutions was that someone could accidentally push a decrypted secret to the git repo. With SOPS when you want to edit a file, the file stays encrypted on disk, gets decrypted in RAM where you can edit it with vim, and when you save the edited file it gets re-encrypted before being written to disk. At the same time, it does offer the flexibility to quickly decrypt a few files so you can use a tool like vimdiff.
SOPS has none of the drawbacks of Vault:
It doesn’t require infrastructure and it’s as cheap as KMS. You could easily set it up, train a few people, and write a readme file within an hour, here’s an example of how easy it is to setup and use:
Bash# aws kms create-key --description "Mozilla SOPS” | grep Arn
Bash# cd $repo
Bash# vim .sops.yaml
(Create a file named .sops.yaml, with the following 2 lines of text)
- kms: 'arn:aws:kms:us-east-1:020522090443:key/4882a19d-5a98-40ae-a1ad-a60423afbddb'
Bash# sops mysecret.yaml
This will open the vim editor so you can type what you want to store in the secret. This simple command is used to both create and edit files.
Bash# cat mysecret.yaml
Will show you an encrypted yaml
Bash# sops --decrypt mysecret.yaml
Will show you the decrypted yaml
SOPS will use your AWS credentials stored in ~/.aws to authenticate against KMS so you can encrypt and decrypt without a password. SOPS will also recursively look for .sops.yaml files so it’ll auto discover metadata about how it should encrypt and decrypt things, which has two important ramifications: first, a user doesn’t have to learn a ton of commands or flags. Second, an additional .sops.yaml file can be added to a subfolder representing a production environment or different project, that .sops.yaml file could have a different encryption/decryption key. You could give different Cloud IAM users different rights to each KMS key, to achieve granular access control. If you’re worried about someone deleting your AWS KMS key you can configure SOPS so the data can be encrypted or decrypted by AWS, GCP, or Azure KMS solutions, so you can keep a secondary backup KMS that few people have access to.
SOPS encourages workflows patterns that make life easier. Devs can store their secrets encrypted right next to and in sync with the version of the code that uses it. Secrets management suddenly gains all the benefits of git: auditable change management, peer reviews via pull requests, diffs of edits to secrets are meaningful because only edited values will get updated on edits, vs the entire file getting re-encrypted, this also makes merge conflicts less likely. Consistency and standardization always make automation and CICD Pipeline development easier, which makes the Ops folks happy. SOPS allows code, configuration, and secrets to be stored in a consistent location, which makes GitOps workflows easier to achieve.
Hashicorp Vault will have trouble achieving its goal of being a centralized secret repo for your organization because users will find it hard to use, devops will find it troublesome to maintain, and management could find it expensive. SOPS, on the other hand, is pain-free to use, easy to learn, cheap to maintain, and supports workflow patterns that make life easier! These things together mean as long as someone can pitch it to the organization, there will be no barriers to adoption, which means an increased security posture for the entire organization is more likely to occur. This is why SOPS with KMS and Git is massively underrated.
I’d like to clarify that the purpose of this article isn’t to say Vault’s bad and you should use SOPS and KMS instead. I wrote this article for 3 reasons: One, I love to teach. Two, I wanted to point out some shortcomings of Vault. Three, KMS with SOPS is an amazing combo that’s massively underrated: no one seems to know about it, I never encountered a proper explanation of either during my research, and according to Google Trends there are not many searches of SOPS compared to Vault.
I’d like to end this article by saying that I wholeheartedly recommend everyone learn SOPS, KMS, and Vault. Why learn Vault if it’s hard and SOPS with KMS does the same thing with ease? Two reasons really: One, Vault is among the best in class when it comes to PKI and secrets rotation, both of which can be needed to satisfy many government and banking security compliance standards. Two, Vault gets easier to use every year: The community has accepted it as a clear winner and added Vault support into several products: Jenkins, cert-manager, and Kubernetes. Kubernetes, in particular, works nicely with Vault, a lot of the pain points have been abstracted and automated to the point where they worth together smoothly. The Vault team also has a proven track record of being committed to making Vault easier to use over time by improving documentation, offering some IaC, and responding to the needs of the community: After the community made auto unseal solutions, backend storage migration solutions, and 3rd party web GUIs; Vault’s Developers decided to bake these functionalities into the open source version. Given this, it wouldn’t surprise me if in the future Vault’s Transit Secrets Engine (Vault’s KMS solution) was made to integrate smoothly with Mozilla SOPS.
Kubernetes Tooling For TechOps And Support (Local Kubernetes Clusters)
Kubernetes tooling for TechOps and Support
Cloud Identity and Access Management – A Customer’s Burden
As passionate technologists, we love to push the envelope. We act as strategists, practitioners and coaches to enable enterprises to adopt modern technology and accelerate innovation.
We help customers win by meeting their business objectives efficiently and effectively.
Join tens of thousands of your peers and sign-up for our best technology content curated by our experts. We never share or sell your email address!
© 2021 Oteemo Inc. All rights reserved