NFS Kerberos in a Kubernetes pod? That’s unpossible!

Recently, I’ve been on a Kubernetes n00b journey and have been documenting the steps here:

This is the Way – My K8s Learning Journey, Part 1: Installing my First K8s Cluster

I’ve also started a new video series called “NetApp Nailed It: Kubernetes Edition” where I encounter problems of my own making and ask experts for help. The first episode can be found here:

I also had written up a blog on using NFS Kerberos in a container. The rub was that I was only able to get it working in a privileged container:

It’s a Kerberos Khristmas!

It was for a customer project and the response was “we need the container to be unprivileged.” I’d been trying to get that working for a while and didn’t see a way to do that. And the end goal was to get this all working within a Google GKE environment. So, my worlds were now colliding!

With some assistance from the good folks at Google, we were able to get this working by using a sidecar pod approach. A privileged pod would do the NFS mount and the data access would be done from the unprivileged pod. This would be accomplished by exposing some paths between the containers and using overlays. Then I added in some customized containers and scripts and it all works pretty seamlessly:

parisi@cloudshell:~$ kubectl create -f privileged-pod.yaml
pod/nfs-poc-privileged created
parisi@cloudshell:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-poc-privileged 1/1 Running 0 6s

parisi@cloudshell:~$ kubectl exec -it nfs-poc-privileged -- configure-nfs.sh
Stopping NFS common utilities: gssd idmapd statd.
Starting NFS common utilities: statd idmapd gssd.
Stopping NFS common utilities: gssd idmapd statd.
Starting NFS common utilities: statd idmapd gssd.
Stopping NFS common utilities: gssd idmapd statd.
Starting NFS common utilities: statd idmapd gssd.
/nfs is not mounted. Mounting /nfs...
Mount success!

parisi@cloudshell:~$ kubectl create -f unpriv-pod.yaml
pod/nfs-poc-unprivileged created
parisi@cloudshell:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-poc-privileged 1/1 Running 0 66s
nfs-poc-unprivileged 1/1 Running 0 3s

parisi@cloudshell:~$ kubectl exec -it nfs-poc-unprivileged -- bash
root@nfs-poc-unprivileged:/# id parisi
uid=1019(parisi) gid=1020(parisigroup) groups=1020(parisigroup),513(domain users)
root@nfs-poc-unprivileged:/# ksu parisi -n parisi
Changing uid to parisi (1019)
bash: /home/parisi/.bashrc: Key has expired
parisi@nfs-poc-unprivileged:/$ kinit
Password for parisi@CVSDEMO.LOCAL:
parisi@nfs-poc-unprivileged:/$ klist
Ticket cache: FILE:/tmp/krb5cc_1019.hnXlZKVV
Default principal: parisi@CVSDEMO.LOCAL

Valid starting Expires Service principal
05/13/22 12:40:32 05/13/22 22:40:32 krbtgt/CVSDEMO.LOCAL@CVSDEMO.LOCAL
renew until 05/20/22 12:40:29

parisi@nfs-poc-unprivileged:/$ cd /home
parisi@nfs-poc-unprivileged:/home$ klist
Ticket cache: FILE:/tmp/krb5cc_1019.hnXlZKVV
Default principal: parisi@CVSDEMO.LOCAL

Valid starting Expires Service principal
05/13/22 12:40:32 05/13/22 22:40:32 krbtgt/CVSDEMO.LOCAL@CVSDEMO.LOCAL
renew until 05/20/22 12:40:29
05/13/22 12:40:40 05/13/22 22:40:32 nfs/nfsserver.cvsdemo.local@CVSDEMO.LOCAL
renew until 05/20/22 12:40:29

You can find the GitHub repository here:

https://github.com/whyistheinternetbroken/k8s-kerberos

Here’s a little more information on what all was needed to make this work properly.

/run/rpc_pipefs mounted

This was needed to ensure all the NFS services could start properly. This is what a listing of that path looks like:

# ls /run/rpc_pipefs
cache gssd lockd mount nfs nfsd nfsd4_cb portmap statd

We did this by exposing the path via a VOLUME command in the container file.

# Attach rpc_pipefs
RUN mkdir -p /run/rpc_pipefs
VOLUME /run/rpc_pipefs

It’s also in the privileged-pod.yaml file:

       - mountPath: /run/rpc_pipefs
         name: rpc-pipefs
         mountPropagation: Bidirectional

An improved configure-nfs.sh script

The previous NFS Kerberos container used a basic script that blindly restarted services on startup. While that works, it isn’t necessary. The new script will check if something is started and then start it. If it’s already started, we’ll just report back that it’s started. And, it also mounts the NFS Kerberos mount in the privileged container. This can be done two ways:

1. Starting the container with bash (the script automatically runs)
2. Starting the container with configure-nfs.sh (the script does all the necessary stuff without needing to enter the container)

There’s likely more improvement possible here, but you can find the script here.

Shared /tmp directory

By default, Kerberos will create krb5cc files when you kinit and place them in /tmp. With two different containers, that /tmp path needed to be shared to allow kinit/Kerberos access to work independently in the unprivileged container. Without this path, Kerberos only worked if you created the Kerberos ticket in the privileged container first. Exposing /tmp allows kinit to work in the unprivileged container without needing to enter the privileged container at all. You can also configure different paths for the krb5cc files in krb5.conf if you choose (including an NFS mount). More details on that cache here:

https://web.mit.edu/kerberos/krb5-1.12/doc/basic/ccache_def.html

Those were the main changes made and everything seems to be humming right along. If you have questions, feel free to leave them in the comments!