CareersContact Us
OpsGuru

Home / Blog / Securing Kubernetes secrets: How to efficiently secure access to etcd and protect your secrets

Securing Kubernetes secrets: How to efficiently secure access to etcd and protect your secrets

Anton Mishel •
Securing Kubernetes secrets: How to efficiently secure access to etcd and protect your secrets

Etcd is a distributed, consistent and highly-available key value store used as the Kubernetes backing store for all cluster data, making it a core component of every K8s deployment.

Due to its central role etcd may contain sensitive information related to access of the deployed services and their associated components, such as database credentials, CA keys, LDAP logins credentials it is a premium target for malicious attacks.

Historically, in traditional, non-containerised environments, this data was NOT stored in such a centralised manner as credentials were usually under an ownership of a specific team that was responsible for maintaining a certain component of the stack: the DB access credentials, for example, were known only to the DBA team, CA keys have been in the hands of few selected System Administrators etc.

With K8s, the required approach is notably different as credentials are now kept within a single central place (etcd), which, if not properly hardened, can lead to serious security breaches as the attacker may now create fake certificates, access databases and applications.

Managing and hardening your secrets becomes even more critical with tools such as Helm and Tiller; these tools allow you to install (or redeploy) an entire K8s based datacenter within minutes and they constantly interact with etcd.

The Center for Internet Security (CIS) came up with this publicly available document providing guidance on how to properly harden and secure your Kubernetes cluster.

The only single recommendation CIS provides regarding hardening etcd is using TLS:

ETCD is a highly available key-value store used by Kubernetes deployments for persistent storage of all of its REST API objects. Its access should be restricted to specifically designated clients and peers only. Authentication to ETCD is based on whether the certificate presented was issued by a trusted certificate authority. There is no checking of certificate attributes such as common name or subject alternative name. As such, if any attackers were able to gain access to any certificate issued by the trusted certificate authority, they would be able to gain full access to the ETCD database. Use a different certificate authority for ETCD from the one used for Kubernetes.

However, using TLS on its own is not sufficient as a solution. Every certificate created and signed with the same CA has the potential to access every service inside the cluster. The problem is further exacerbated if a single CA is used for all k8s clusters. Even when each kubernetes cluster has a dedicated CA, new client keys can be easily created but as easily revoked. Once again, any new keys created automatically have access to every service in the targeted k8s cluster.

Because of the severity of the security risks associated with etcd, we will look into 2 additional methods that can be implemented to further secure your etcd data:

Encrypting secrets (and/or other resources) in etcdUsing certificates to stop clients from accessing the etcd server

To follow the steps illustrated in the following sections, it is necessary to start up a Kubernetes cluster. This can be done using any of the methods immediately below.

Vanilla K8s:

install script for latest kubernetes 1.10. This is the first version that installs etcd with tls install script for older kubernetes versions, when etcd was not installed with tls by default

This also works on openshift platform. You can see the install script here. All relative commands for openshift are in this script.

The install scripts have been tested on AWS Centos ami. It should work for you too if you use the same image.

Encrypting Secret Data at Rest

Starting with K8s 1.7 (and etcd v3) you can encrypt resources inside etcd using several different algorithms. At the very least, you should encrypt all your secrets. It is especially true if you are using Helm as a lot of Helm charts require LDAP or DB credentials to be directly made available in the ConfigMaps.

The encryption follows a very simple rule:

encrypt using the first provider defineddecrypt after locating a functional provider at checking each provider in the order the providers are defined

To implement the full workflow, it is necessary to add the experimental-encryption-provider-config flag to the apiserver

Define the EncryptionConfig config file (place the content in /etc/kubernetes/pki/encryption-config.yaml)

kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
    - secrets
    providers:
    - identity: {}

Within the file, the resources.resources field is an array of Kubernetes resource names that should be encrypted. The providers array is an ordered list of the possible encryption providers.

Enable experimental-encryption-provider-config in the kube-apiserver. Edit /etc/kubernetes/manifests/kube-apiserver.yaml and add:

spec:
  containers:
  - command:
    - kube-apiserver
    - --experimental-encryption-provider-config=/etc/kubernetes/pki/encryption-config.yaml

Restart the apiserver. Because the API server is being run as a static pod, kubelet will restart it when the configuration change is detected. Otherwise, you will need to restart the service yourself.

We also install the etcd package in order to print the data from inside the etcd server:

# yum install etcd -y
Resolving Dependencies
--> Running transaction check
---> Package etcd.x86_64 0:3.2.18-1.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================================================================================================================================
 Package                                        Arch                                             Version                                                   Repository                                                        Size
==================================================================================================================================================================================================================================
Installing:
 etcd                                           x86_64                                           3.2.18-1.el7                                              optymyze_external_rpms                                           9.3 M
Transaction Summary
==================================================================================================================================================================================================================================
Install  1 Package
Total download size: 9.3 M
Installed size: 42 M
Downloading packages:
etcd-3.2.18-1.el7.x86_64.rpm                                                                                                                                                                               | 9.3 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : etcd-3.2.18-1.el7.x86_64                                                                                                                                                                                       1/1 
Uploading Package Profile
  Verifying  : etcd-3.2.18-1.el7.x86_64                                                                                                                                                                                       1/1
Installed:
  etcd.x86_64 0:3.2.18-1.el7
Complete!
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, priorities, product-id

Also, to make the commands shorter, set an alias for etcdctl command with TLS parameters. Here we will use the certificates paths created by kubeadm-1.10. You should update them for your specific cluster if needed (check by runninggrep -- '--etcd' /etc/kubernetes/manifests/kube-apiserver.yaml).

Meaning of variables:

DIR — path where the k8s certificates are createdSSLOPS — etcdctl parameters to enable TLS connectivitySECRETSPATH — path in etcd where kubernetes keeps secrets

DIR=/etc/kubernetes/pki/
SSL_OPTS="--cacert=${DIR}/etcd/ca.crt --cert=${DIR}/apiserver-etcd-client.crt --key=${DIR}/apiserver-etcd-client.key --endpoints=localhost:2379"
SECRETS_PATH=/registry/secrets

Test that we can list stuff in etcd:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only=true --prefix $SECRETS_PATH
/registry/secrets/default/default-token-rhwwn
/registry/secrets/kube-public/default-token-9qfc8
/registry/secrets/kube-system/attachdetach-controller-token-clvsn
.............

No Encryption

To demonstrate the difference of our solution, we begin with no encryption. This provider doesn’t do any encryption. It can be used in case you want to decrypt everything or just to test.

Let’s create a secret and read it directly from etcd. You should be able to clearly see the key name and key value:

# kubectl create secret generic secret1  --from-literal=XX_mykey_XX=ZZ_mydata_ZZ
secret "secret1" created
# kubectl get secret secret1 -o yaml
apiVersion: v1
data:
  XX_mykey_XX: WlpfbXlkYXRhX1pa
kind: Secret
metadata:
  creationTimestamp: 2018-06-18T13:11:54Z
  name: secret1
  namespace: default
  resourceVersion: "20410585"
  selfLink: /api/v1/namespaces/default/secrets/secret1
  uid: 2bb3b7df-72f9-11e8-ad5f-005056b1028d
type: Opaque
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value
"Value" : "k8s\x00\n\f\n\x02v1\x12\x06Secret\x12s\nL\n\asecret1\x12\x00\x1a\adefault\"\x00*$2bb3b7df-72f9-11e8-ad5f-005056b1028d2\x008\x00B\b\b\x9aߞ\xd9\x05\x10\x00z\x00\x12\x1b\n\vXX_mykey_XX\x12\fZZ_mydata_ZZ\x1a\x06Opaque\x1a\x00\"\x00"

Apply an Encryption Algorithm

Let’s add an encryption algorithm to see what happens. We choose aescbcbecause this is the recommended choice for encryption at rest.

Update encryption-config.yaml:

kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: c2VjcmV0IGlzIHNlY3VyZQ==
        - name: key2
          secret: dGhpcyBpcyBwYXNzd29yZA==
    - identity: {}

Since kubelet only monitors pods defined in /etc/kubernetes/manifests, this change will not be caught, so we need to restart the apiserver manually:

docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')

Test:

# kubectl create secret generic secret2 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ
secret "secret2" created
# kubectl get secret secret2 -o yaml
apiVersion: v1
data:
  XX_mykey_XX: WlpfbXlkYXRhX1pa
kind: Secret
metadata:
  creationTimestamp: 2018-06-18T14:23:06Z
  name: secret2
  namespace: default
  resourceVersion: "20418382"
  selfLink: /api/v1/namespaces/default/secrets/secret2
  uid: 1e4f5d2f-7303-11e8-8c2c-005056b1028d
type: Opaque
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value
"Value" : "k8s:enc:aescbc:v1:key1:7^İ\xe9\xc8\x1e\xa7̔=D+\x9e%\x1a\xf4\x10o@\xec\xc14&<Z\xd1\xde\xfa\xca-'#\xa2K\x1c\xff\x101a\x86\xb0\xd7.\xa9\x19\x04\x93m\xa1\xee\xacDe\x95/\xd8\xe7\xaehp~\xc9\x0e\xe9\x8f}\x9a\x8a\xb0f\xf9\xeb\xb7\u007f@\x87\xa0\xa6\x98\xe78\xd0+\xd45\"S\x17\x8c\x84\xa6ㅽb\xda\xe6\xfc\xa1\xd9[[~\x82\xfbKS\x82\xf0>o\xc1 \x8b&{\xa1\r\x14Un\x03\xf7\x1f=\xe5\x1b \xa7t\xed[\x8a\xec\xb8\xf1\xe4\xe2\xc1\x81\xb00=cbl·ɬ\x12`\xf2|\x1b\t\xe4#\xcd"

The new secret was encrypted now with “k8s:enc:aescbc:v1:key1”.

Let’s encrypt all the other secrets as well:

# kubectl get secrets --all-namespaces -o json | kubectl replace -f -
secret "default-token-rhwwn" replaced
secret "secret1" replaced
secret "secret2" replaced
secret "default-token-9qfc8" replaced
secret "attachdetach-controller-token-clvsn" replaced
secret "bootstrap-signer-token-xgnfg" replaced
....

Check that the old secret is now encrypted:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value
"Value" : "k8s:enc:aescbc:v1:key1:\xda\nW0~\x83\xe4\x80Ճ$J\x1e\xa2\x02z\xc9\v\xd1\xd0$)\xb2K\x9f\xc2\xff\xcdJ5\xfa\"\x13\xc4\f\x86\xc0{P\xceW\x9e\xd1z;b$\x97\xe8\xb4l\xd0\xfa\xd8 \xe2Vc\x8c\xa2\xcd\xe5\xb0\x04(l\x18\x13\xbf\xe2\xb7|\xf1m\xef)\xfd\x97\xcbk-\"\xba\x819\xcf,_\xf6\fxP\xf2\x13\x94\x9b\xca\xf4\xde{d\xcb\xceq\x84q\xae\xaa\x06\x14\xb7q\x1d|L\x8eS\x8c\xc9$\x8e\x80D\xf0\xda\xe2si\xb6,@\xa2\xf9\xae\xf2~\xe3w\x8e4fr{e\x0f'\xcc\xf6\xe7\xadd\x83^\xdb\x03\xf1jT\x13>"

Each resource is encrypted with a specific key. If you change the value of that key, kubernetes will not be able to decode it anymore. You can test this by swapping the name of the keys and try to retrieve it:

# cat /etc/kubernetes/pki/encryption-config.yaml 
kind: EncryptionConfig
apiVersion: v1
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key2
          secret: c2VjcmV0IGlzIHNlY3VyZQ==
        - name: key1
          secret: dGhpcyBpcyBwYXNzd29yZA==
    - identity: {}
# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')
000e03b50c0f
# kubectl get secret secret2 -o yaml
Error from server (InternalError): Internal error occurred: invalid PKCS7 data (empty or not padded)

Using multiple algorithms

In this example we encrypt a secret with a new algorithm and check that different secrets are encrypted with different providers. After that encrypt everything with the new provider. This will change the encryption algorithm to all previous keys to the new one.

Change encryption-config.yaml, so that the first provider to be the secretbox provider:

kind: EncryptionConfig
apiVersion: v1
resources: 
  - resources: 
    - secrets
    providers:
    - secretbox:
        keys:
        - name: key1
          secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
    - aescbc:
        keys:
        - name: key1
          secret: c2VjcmV0IGlzIHNlY3VyZQ==
        - name: key2
          secret: dGhpcyBpcyBwYXNzd29yZA==
    - identity: {}

Restart the apiserver:

# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')

Verify that secrets are encrypted correctly: old secret is using aescbc, new one will use secretbox.

# kubectl create secret generic secret3 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ
secret "secret3" created
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value
"Value" : "k8s:enc:aescbc:v1:key1:\xda\nW0~\x83\xe4\x80Ճ$J\x1e\xa2\x02z\xc9\v\xd1\xd0$)\xb2K\x9f\xc2\xff\xcdJ5\xfa\"\x13\xc4\f\x86\xc0{P\xceW\x9e\xd1z;b$\x97\xe8\xb4l\xd0\xfa\xd8 \xe2Vc\x8c\xa2\xcd\xe5\xb0\x04(l\x18\x13\xbf\xe2\xb7|\xf1m\xef)\xfd\x97\xcbk-\"\xba\x819\xcf,_\xf6\fxP\xf2\x13\x94\x9b\xca\xf4\xde{d\xcb\xceq\x84q\xae\xaa\x06\x14\xb7q\x1d|L\x8eS\x8c\xc9$\x8e\x80D\xf0\xda\xe2si\xb6,@\xa2\xf9\xae\xf2~\xe3w\x8e4fr{e\x0f'\xcc\xf6\xe7\xadd\x83^\xdb\x03\xf1jT\x13>"
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value
"Value" : "k8s:enc:secretbox:v1:key1:\xba\xf8,Q@\xb9\xb6q3$k\x04\xeeV\x99|Z'\xdeE<\xa5\xa9n\x91u\xb9]RY\xccc\xe3\x13\x8b\u07b4Q\x91\x9cR2\xcc\xc5\xd9\x0e\x19?\xca\x1ch\xde\x1d%\xa3N\x85H\xb0\xf6֢\xe6\xab\x06\xf6\x960{\xdb\xd8^eQ\xb3\x05\x03\x06)\x05JH\x16\x18\fp\x9eu<t\xea\x06\x12\xf1۹y\u007f\x15\xe5\x1d\xef\x8a2G\x85'\x94\n\x1d\x99\x85ku3\xa2~\x12\x04\xe5\x84~\xaaG\xd3n\x98\x95\xa0\xc8_1B\xcb\x0f\xb7;\x80\xe1xR\x86ij\f\xef\xd7SA\x950MQfz~)\x13\xc5\xf1\xf8\x91\x14\x9d_\xba\x82[=M\x81O\x1dFNj\xc1\x98\xe4"

Migrate all secrets to the new provider:

# kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Key rotation

Here we use the same provider for encryption, but we add a new key. Everything from this moment will be encrypted with the new key. Old values are encrypted with the previous key. At the end we migrate everything to be encrypted with the new key.

Add a new key to be the first for the secretbox provider (which is still the first provider)

kind: EncryptionConfig
apiVersion: v1
resources: 
  - resources: 
    - secrets
    providers:
    - secretbox:
        keys:
        - name: key2
          secret: sAkccgM28JdPNCX9FfTcloYet1zp4OEAtHyViT038zM=
        - name: key1
          secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
    - aescbc:
        keys:
        - name: key1
          secret: c2VjcmV0IGlzIHNlY3VyZQ==
        - name: key2
          secret: dGhpcyBpcyBwYXNzd29yZA==
    - identity: {}

Restart and verify that the new values are encrypted with the new key:

# docker stop $(docker ps | grep k8s_kube-apiserver | gawk '{print $1}')
4bdac1937570
# kubectl create secret generic secret4 --from-literal=XX_mykey_XX=ZZ_mydata_ZZ
secret "secret4" created
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value
"Value" : "k8s:enc:secretbox:v1:key2:\x92\xeeyj\x96\xfc쵪-8\x0e\xa7\x9a\xb0\x16\xe2\xb8J\f_\x81\xec\xf65\xa9\x1a\xe5\\xۛ%Ҝ\xbb\ax\xbf\x00Kz\xabaD\x1c\x94\x87\xaervsP\xf3q\xf3\xaeH\xb8\x95-\xef\r*[yl\xf3/\xc4\x0f\x00\a\x132\f\xe1\x17\xbf\xff\xb4;<\xec\xc2\x01\xa8\xc8f\xff\xcd\xf3ʦ\x83P\x01\xcdu\x16\x16\xfa\xba\x8f\xe6\xe5\x05\x96\xf7k,\xaa\xea\x0f\x99\x8f\xb3\xc7\xe6\xa4=\x93\x8a\xf3S\x17\xc6S\r\xee\xea㟷\x00\x945o\xe8\x8e:W\xacot\xeaj,P\x14\xbe\xd0\x13\xf91Y\xf0\xf0\x93fW\xcczD3\xb9\xa0\xb4\x9e\xef\x1aE\x16\xc8j_TX\xae"

ETCD authorization

Etcd can use 2 methods to authorize users:

With username and passwordWith certificates if started with ”--client-cert-auth=true”. It will use the CN from the certificate as the username.

Unfortunately there are a couple of issues with this:

https://github.com/coreos/etcd/issues/9816: auth doesn’t work at all with the default etcd version. You need to update your etcd to version 3.2.18https://github.com/coreos/etcd/issues/9691: this affects you if you try to create the openshift user inside etcd. But for this one there is also an workaround

We use cfssl to create the certificates we need to connect to etcd. Lets download the binaries and put them in our path:

# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O && /bin/mv ./cfssl_linux-amd64 /bin/cfssl && chmod +x /bin/cfssl
# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O && /bin/mv ./cfssljson_linux-amd64 /bin/cfssljson && chmod +x /bin/cfssljson

In order to enable etcd ACL, first create the root user:

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add root:secretpass
User root created

Create some roles to test our certificates.

Users with this role should have read access in the entire cluster:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_all
Role readonly_all created
# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_all --prefix=true read /
Role readonly_all updated

Allows users to read and write everywhere:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_all
Role readwrite_all created
# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_all --prefix=true readwrite /
Role readwrite_all updated

User with this role should only be allowed to access part of etcd tree:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readonly_secrets
Role readonly_secrets created
# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readonly_secrets --prefix=true read $SECRETS_PATH
Role readonly_secrets updated

Role that allows a user to read/write a specific key only:

# ETCDCTL_API=3 etcdctl $SSL_OPTS role add readwrite_secret4
Role readwrite_secret4 created
# ETCDCTL_API=3 etcdctl $SSL_OPTS role grant-permission readwrite_secret4 readwrite $SECRETS_PATH/default/secret4
Role readwrite_secret4 updated

Create users and add assign specific roles to them. Generate random passwords because we don’t expect to use them:

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add reader:$(head -c 32 /dev/urandom | base64)
User reader created
# ETCDCTL_API=3 etcdctl $SSL_OPTS user add viewsecrets:$(head -c 32 /dev/urandom | base64)
User viewsecrets created
# ETCDCTL_API=3 etcdctl $SSL_OPTS user add admin:$(head -c 32 /dev/urandom | base64)
User admin created
# ETCDCTL_API=3 etcdctl $SSL_OPTS user add usersecret4:$(head -c 32 /dev/urandom | base64)
User usersecret4 created
# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role reader readonly_all
Role readonly_all is granted to user reader
# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role viewsecrets readonly_secrets
Role readonly_secrets is granted to user viewsecrets
# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role admin readwrite_all
Role readwrite_all is granted to user admin
# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role usersecret4 readwrite_secret4
Role readwrite_secret4 is granted to user usersecret4

Since we are enabling user authorization, we need to have special permissions for the user used by the apiserver to connect to the etcd cluster: we will give it the root role. Kubernetes installation has 2 connections to the etcd server: apiserver and the livenessProbe.

openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver-etcd-client.crt | grep "Subject:"
openssl x509 -noout -text -in /etc/kubernetes/pki/etcd/healthcheck-client.crt | grep "Subject:"

Create an user with the name from the CN field of the certificate:

# ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-apiserver-etcd-client:$(head -c 32 /dev/urandom | base64)
User kube-apiserver-etcd-client created
# ETCDCTL_API=3 etcdctl $SSL_OPTS user add kube-etcd-healthcheck-client:$(head -c 32 /dev/urandom | base64)
User kube-etcd-healthcheck-client created
# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-apiserver-etcd-client root
Role root is granted to user kube-apiserver-etcd-client
# ETCDCTL_API=3 etcdctl $SSL_OPTS user grant-role kube-etcd-healthcheck-client root
Role root is granted to user kube-etcd-healthcheck-client

Enable authentication:

# ETCDCTL_API=3 etcdctl $SSL_OPTS auth enable
Authentication Enabled

From this moment, nothing can connect to the etcd cluster without proper certificates. Let’s create a certificate with a user that it’s not define in etcd and check that it doesn’t have access at all:

# function create_certificates {
  NAME=$1
  
  cat <<EOF | cfssl gencert -config=ca-config.json -profile=client -ca $CA_PATH/ca.crt -ca-key $CA_PATH/ca.key - | cfssljson -bare $NAME
{"CN": "$NAME","key": {"algo": "rsa","size": 2048}}
EOF
SSL_OPTS="--cacert=$CA_PATH/ca.crt --cert=$PWD/$NAME.pem --key=$PWD/$NAME-key.pem --endpoints=$HOSTNAME:2379"
}
# CA_PATH=/etc/kubernetes/pki/etcd
# cfssl print-defaults config > ca-config.json
# create_certificates tester
2018/06/18 18:52:54 [INFO] generate received request
2018/06/18 18:52:54 [INFO] received CSR
2018/06/18 18:52:54 [INFO] generating key: rsa-2048
2018/06/18 18:52:54 [INFO] encoded CSR
2018/06/18 18:52:54 [INFO] signed certificate with serial number 722235405009026418318053946143102861163105227800
2018/06/18 18:52:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
# ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true
Error:  etcdserver: permission denied

Allow an admin user to access the cluster for 2 hours only:

# cfssl print-defaults config | sed s/8760/2/ > ca-config.json
# create_certificates admin

Admin user can do list:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get / --keys-only --prefix=true
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io

Get a key:

# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret1 -w fields | grep Value
"Value" : "k8s:enc:aescbc:v1:key1:\xda\nW0~\x83\xe4\x80Ճ$J\x1e\xa2\x02z\xc9\v\xd1\xd0$)\xb2K\x9f\xc2\xff\xcdJ5\xfa\"\x13\xc4\f\x86\xc0{P\xceW\x9e\xd1z;b$\x97\xe8\xb4l\xd0\xfa\xd8 \xe2Vc\x8c\xa2\xcd\xe5\xb0\x04(l\x18\x13\xbf\xe2\xb7|\xf1m\xef)\xfd\x97\xcbk-\"\xba\x819\xcf,_\xf6\fxP\xf2\x13\x94\x9b\xca\xf4\xde{d\xcb\xceq\x84q\xae\xaa\x06\x14\xb7q\x1d|L\x8eS\x8c\xc9$\x8e\x80D\xf0\xda\xe2si\xb6,@\xa2\xf9\xae\xf2~\xe3w\x8e4fr{e\x0f'\xcc\xf6\xe7\xadd\x83^\xdb\x03\xf1jT\x13>"

Delete the key:

# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret1
1
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/secret1 -w fields | grep Value

Increase the date and try to list:

# date $(date +%m%d%H%M%Y.%S -d '+1 hour')
# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
# date $(date +%m%d%H%M%Y.%S -d '+1 hour')
# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /
Error:  context deadline exceeded
# date $(date +%m%d%H%M%Y.%S -d '-2 hour')

Check that user ‘reader’ has access everywhere and can’t delete anything:

# cfssl print-defaults config > ca-config.json
# create_certificates reader
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret2 -w fields | grep Value
"Value" : "k8s:enc:aescbc:v1:key1:7^İ\xe9\xc8\x1e\xa7̔=D+\x9e%\x1a\xf4\x10o@\xec\xc14&<Z\xd1\xde\xfa\xca-'#\xa2K\x1c\xff\x101a\x86\xb0\xd7.\xa9\x19\x04\x93m\xa1\xee\xacDe\x95/\xd8\xe7\xaehp~\xc9\x0e\xe9\x8f}\x9a\x8a\xb0f\xf9\xeb\xb7\u007f@\x87\xa0\xa6\x98\xe78\xd0+\xd45\"S\x17\x8c\x84\xa6ㅽb\xda\xe6\xfc\xa1\xd9[[~\x82\xfbKS\x82\xf0>o\xc1 \x8b&{\xa1\r\x14Un\x03\xf7\x1f=\xe5\x1b \xa7t\xed[\x8a\xec\xb8\xf1\xe4\xe2\xc1\x81\xb00=cbl·ɬ\x12`\xf2|\x1b\t\xe4#\xcd"
# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret2
Error:  etcdserver: permission denied

Check that user ‘viewsecrets’ has access only to read secrets:

# create_certificates viewsecrets
# ETCDCTL_API=3 etcdctl $SSL_OPTS get --keys-only --prefix=true /
Error:  etcdserver: permission denied
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value
"Value" : "k8s:enc:secretbox:v1:key1:\xba\xf8,Q@\xb9\xb6q3$k\x04\xeeV\x99|Z'\xdeE<\xa5\xa9n\x91u\xb9]RY\xccc\xe3\x13\x8b\u07b4Q\x91\x9cR2\xcc\xc5\xd9\x0e\x19?\xca\x1ch\xde\x1d%\xa3N\x85H\xb0\xf6֢\xe6\xab\x06\xf6\x960{\xdb\xd8^eQ\xb3\x05\x03\x06)\x05JH\x16\x18\fp\x9eu<t\xea\x06\x12\xf1۹y\u007f\x15\xe5\x1d\xef\x8a2G\x85'\x94\n\x1d\x99\x85ku3\xa2~\x12\x04\xe5\x84~\xaaG\xd3n\x98\x95\xa0\xc8_1B\xcb\x0f\xb7;\x80\xe1xR\x86ij\f\xef\xd7SA\x950MQfz~)\x13\xc5\xf1\xf8\x91\x14\x9d_\xba\x82[=M\x81O\x1dFNj\xc1\x98\xe4"
# ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret3
Error:  etcdserver: permission denied

Check that ‘usersecret4’ can access only a specific key:

# create_certificates usersecret4
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret3 -w fields | grep Value
Error:  etcdserver: permission denied
# ETCDCTL_API=3 etcdctl $SSL_OPTS get $SECRETS_PATH/default/secret4 -w fields | grep Value
"Value" : "k8s:enc:secretbox:v1:key2:\x92\xeeyj\x96\xfc쵪-8\x0e\xa7\x9a\xb0\x16\xe2\xb8J\f_\x81\xec\xf65\xa9\x1a\xe5\\xۛ%Ҝ\xbb\ax\xbf\x00Kz\xabaD\x1c\x94\x87\xaervsP\xf3q\xf3\xaeH\xb8\x95-\xef\r*[yl\xf3/\xc4\x0f\x00\a\x132\f\xe1\x17\xbf\xff\xb4;<\xec\xc2\x01\xa8\xc8f\xff\xcd\xf3ʦ\x83P\x01\xcdu\x16\x16\xfa\xba\x8f\xe6\xe5\x05\x96\xf7k,\xaa\xea\x0f\x99\x8f\xb3\xc7\xe6\xa4=\x93\x8a\xf3S\x17\xc6S\r\xee\xea㟷\x00\x945o\xe8\x8e:W\xacot\xeaj,P\x14\xbe\xd0\x13\xf91Y\xf0\xf0\x93fW\xcczD3\xb9\xa0\xb4\x9e\xef\x1aE\x16\xc8j_TX\xae"
$ ETCDCTL_API=3 etcdctl $SSL_OPTS del $SECRETS_PATH/default/secret4
1

Conclusion

In a cluster with multiple masters, where etcd servers listen on all interfaces and not on localhost, limiting the access to etcd is vital.
Kubernetes manages this with RBAC, but by default etcd is only protected by requiring the client to have a valid certificate.
While employing EncryptionConfig can take care of most of the issues, it is still possible to have data in etcd that is not fully encrypted. Since confidential data can now be required in configmaps, even in statefulsets and deployments as environment variables, using simply EncryptionConfig is not sufficient.
If you cut access entirely to etcd by using authentication and only allow the apiserver to connect, you protect yourself from leaking sensitive data to others. You can create short lived certificates for any other uses and no longer need to worry about the longevity of the certificates.
The solution is also applicable to the users added previously to etcd: if you don’t feel confident about what happened to certificates created for them before, you can revoke the users access, or delete them.

All Tags

KubernetesSecurity

Share the Blog

Related Posts

Contact UsLearn more about how OpsGuru can help you with cloud adoption through our cloud services.