In this post we will address the creation and usage of wild-card certificates in our Kubernetes cluster using cert-manager and nginx-ingress. This article is intended for people with some base understanding of Kubernetes, Cert-manager, and Nginx. That being said, Cert-manager is an awesome tool that automatically negotiate’s SSL certificates on our behalf using external providers like Let’s Encrypt. The Nginx-ingress-controller is a tool that allows you to configure a HTTP load balancer to expose your Kubernetes services outside of your cluster. We will be installing nginx-ingress and cert-manager with a Kubernetes package management tool called helm. Once these tools are installed we will create a cluster-issuer for cert-manager and a wild-card certificate. The wild-card certificate secret will only be created in a single namespace. So we will then use a tool call kubed to replicate the secret across multiple namespaces. To spice things up instead of manually annotating the secret for kubed, we will create a kubernetes cronjob to annotate it for us.
Install Helm
First thing’s first lets install Helm client tool. For mac:
$ brew install kubernetes-helm
Other Operating Systems refer to: https://docs.helm.sh/using_helm/#installing-helm Since I have RBAC enabled on my cluster, we must deploy a service account and cluster role/binding for the tiller pod to use.
$ cat <
Deploy tiller pod in your cluster
helm init --service-account tiller
Install Nginx Ingress Controller
Using helm we will install an nginx ingress controller to be able to forward external traffic to the correct kubernetes resources in our cluster.
$ helm install --name nginx stable/nginx-ingress
Install Cert-manager
Next we will install cert-manager. Cert-manager is an awesome tool that reaches out to external certificate providers like Let’s Encrypt and generates valid TLS certificates for us.
NOTE: Be sure to configure the correct permissions to run cert-manager in the cloud provider you’re using, for more info refer too: https://github.com/jetstack/cert-manager/blob/master/docs/reference/issuers/acme/dns01.rst
$ helm install --name cert-manager stable/cert-manager
Make Sure everything is up and running
List installed charts
helm ls
Expected Output:
NAME REVISION UPDATED STATUS CHART NAMESPACE
cert-manager 1 Tue Sep 4 23:20:02 2018 DEPLOYED cert-manager-v0.4.1 default
nginx 1 Tue Sep 4 23:19:54 2018 DEPLOYED nginx-ingress-0.26.0 default
Make sure pods are up and running
kubectl get pods
Expected Output:
NAME READY STATUS RESTARTS AGE
cert-manager-756d6d885d-6nxz7 1/1 Running 0 48s
nginx-nginx-ingress-controller-5b79ff75b6-p4zzn 1/1 Running 0 57s
nginx-nginx-ingress-default-backend-db4db5d6d-7c9bp 1/1 Running 0 57s
Make sure load balancer has been deployed for nginx
kubectl get svc -l component=controller
Expected Output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-nginx-ingress-controller LoadBalancer 80:30050/TCP,443:31903/TCP 2m
Install Cluster-Issuers
In order for cert-manager to be able to create certificates we must configure a ClusterIssuer. This tells cert-manager information about what certificate provider we would like to use, and what domain validation technique to use.
$ cat <
Install Wild-Card Certificate
Next lets create the CRD which tells the clusterissuer to go get a cert from let’s encrypt using DNS as the validation
$ cat <
Next lets tail the logs for cert-manager to see what is going on.
kubectl logs -f cert-manager-756d6d885d-6nxz7
If everything has been configured correctly you should see cert manager pick up the wild card certificate, reach out to Lets Encrypt, validate your domain, and finally issue a certificate. Once the certificate has been issued by Lets Encrypt you should now see a secret called wildcard-certificate.
NOTE: Again if you see a “not authorized” or “permission denied” make sure you had cert-manager permissions configured correctly.
kubectl get secrets
Expected Output:
NAME TYPE DATA AGE
cert-manager-token-z6c5f kubernetes.io/service-account-token 3 10m
default-token-24pf2 kubernetes.io/service-account-token 3 10m
letsencrypt-prod Opaque 1 10m
nginx-nginx-ingress-token-7nk2t kubernetes.io/service-account-token 3 10m
wildcard-certificate kubernetes.io/tls 2 3m
Install Kubed
So far we have created a secret containing the wild card certificate in only one namespace (default). If we wish to deploy ingress’s that use this TLS certificate in other namespaces we must some how replicate this secret into other namespaces. We will do this with the help of a tool called kubed.
$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ helm install appscode/kubed --name kubed --namespace kube-system
--set apiserver.enabled=false
--set config.clusterName=prod_cluster
Make sure kubed is up and running
$ kubectl get pods --namespace kube-system -l app=kubed
Expected Output:
NAME READY STATUS RESTARTS AGE
kubed-kubed-79d887dd5d-mxfgq 1/1 Running 0 1m
Annotate Secrets
Now that kubed is up and running lets annotate the secret we want to replicate across multiple namespaces.
$ kubectl annotate secret wildcard-certificate kubed.appscode.com/sync="app=kubed"
Expected Output:
secret " wildcard-certificate" annotated
Create a Labeled Namespace
Next we will create a new namespace. This new namespace will have a label on it to let kubed now we want to replicate the secret here.
$ cat <
Make sure the namespace was created:
$ kubectl get namespaces
Expected Output:
NAME STATUS AGE
default Active 1h
demo-namespace Active 8s
kube-public Active 1h
kube-system Active 1h
Kubed picks up that a new namespace has been created with the label app=kubed so it replicates that secret into this new namespace. Make sure the secret has been added to the new namespace:
kubectl get secrets --namespace demo-namespace
Expected Output:
NAME TYPE DATA AGE
default-token-264z2 kubernetes.io/service-account-token 3 19s
wildcard-certificate kubernetes.io/tls 2 19s
Create Cronjob
So far so good. We have * deployed nginx, cert-manager, and kubed * created a cluster-issuer and wild-card certificate * waited for Lets Encrypt to issue us a TLS certificate * annotated this new secret to replicate it across it multiple namespaces However, what if we wanted to automate this process? As you noticed cert-manager needed to wait for Lets Encrypt to validate the domain before creating a secret with the TLS certificate. So let’s write a kubernetes cron-job to annotate the secret for us when it is created.
Bash
Lets throw together a quick bash script to annotate the secrets for us. Our script expects a list containing the names of the secrets we want to annotate.
#!/bin/bash
IFS=', ' read -r -a SECRETS_LIST <<< "${SECRETS_LIST}"
#Function to check if a given secret exists
#Parameters: ${1}=Namespace ${2}=Secret Name
secret_exists() {
/usr/local/bin/kubectl get secrets -n $1 |
grep $2
&>/dev/null
echo "$?"
}
#Function to check if a given label is on a secret
# Parameters: ${1}=Namespace ${2}=Secret Name ${3}=Annotation
annotation_exists() {
/usr/local/bin/kubectl get secrets -n ${1} ${2} -o=jsonpath='{.metadata.annotations}' |
grep kubed.appscode.com/sync:app=kubed
&>/dev/null
echo "$?"
}
#Function to add annotation to secret
#Parameters: ${1}=Namespace ${2}=Secret Name
add_annotation() {
/usr/local/bin/kubectl annotate secret -n ${1} ${2} kubed.appscode.com/sync="app=kubed"
&>/dev/null
echo "$?"
}
echo ${SECRETS_LIST[@]}
COUNTER=0
for item in "${SECRETS_LIST[@]}"
do
echo "Working on: ${item}"
echo "Checking if secret exists..."
secret_res=$(secret_exists ${SECRETS_NAMESPACE} ${item})
if [ ${secret_res} = 0 ];
then
echo "Secret Exists!"
echo "Checking if annotation exists..."
anno_res=$(annotation_exists ${SECRETS_NAMESPACE} ${item})
echo "result=${anno_res}"
if [ ${anno_res} = 0 ];
then
echo "Annotation exists!"
let COUNTER=COUNTER+1
else
echo "Annotation does not exist"
echo "Adding Annotation..."
add_anno_res=$(add_annotation ${SECRETS_NAMESPACE} ${item})
fi
else
echo "Secret does not exist"
fi
done
if [ ${COUNTER} = ${#SECRETS_LIST[*]} ];
then
echo "All secrets are annotated"
fi
Configmap
We will mount this bash script into our cronjob using a configmap, so lets go ahead and copy this code into a file called cronjob.sh and then create a configmap.
$ kubectl create configmap wild-card-certs-cronjob --from-file=cronjob.sh
Expected Output
configmap/wild-card-certs-cronjob created
Cronjob
Our cronjob.yaml will use a docker image that has kubectl already installed.
$ cat <
Expected Output:
cronjob.batch/wild-card-certs-cronjob created
Service Account, ClusterRole, ClusterRoleBinding
Since we are using RBAC in our cluster we must create a service account, cluster role and role binding for our cronjob to use. This will allow our cronjob to be able to manipulate secrets, and cronjobs with in our cluster.
$ cat <
Use Wild-card Secret in ingress
Lets plop this secret in an ingress and see it in action!
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
name:
namespace: demo1
spec:
rules:
- host:
http:
paths:
- backend:
serviceName:
servicePort: 80
path: /
tls:
- hosts:
-
secretName: wildcard-certificate
If everything works as expected you should be able to now visit your website and see that is being served over https! To further inspect the certificate from the client side we can run the following curl command to get and inspect the certificate. echo |
openssl s_client -showcerts -servername
-connect :443 2>/dev/null |
openssl x509 -inform pem -noout -text
Conclusion
We have just learned how to automate, the negotiation and creation, of wild card certificates using cert-manager, and creating an ingress into our cluster using nginx. Finally in order to replicate the secrets created by cert-manager to multiple namespaces we have used a tool called kubed. Armed with this knowledge we should be able to easily deploy applications protected by SSL! References: https://rimusz.net/lets-encrypt-wildcard-certs-in-kubernetes/
Oteemo thanks you for reading this post, if you want to find out more as to DoD Iron Bank services, DevSecOps adoption or Cloud Native Enablement, contact us by submitting our form via our website.
We will be happy to answer all your questions, especially the hard ones!