Recently, I’ve been on a Kubernetes n00b journey and have been documenting the steps here:
This is the Way – My K8s Learning Journey, Part 1: Installing my First K8s Cluster
I’ve also started a new video series called “NetApp Nailed It: Kubernetes Edition” where I encounter problems of my own making and ask experts for help. The first episode can be found here:
I also had written up a blog on using NFS Kerberos in a container. The rub was that I was only able to get it working in a privileged container:
It was for a customer project and the response was “we need the container to be unprivileged.” I’d been trying to get that working for a while and didn’t see a way to do that. And the end goal was to get this all working within a Google GKE environment. So, my worlds were now colliding!
With some assistance from the good folks at Google, we were able to get this working by using a sidecar pod approach. A privileged pod would do the NFS mount and the data access would be done from the unprivileged pod. This would be accomplished by exposing some paths between the containers and using overlays. Then I added in some customized containers and scripts and it all works pretty seamlessly:
parisi@cloudshell:~$ kubectl create -f privileged-pod.yaml pod/nfs-poc-privileged created parisi@cloudshell:~$ kubectl get pods NAME READY STATUS RESTARTS AGE nfs-poc-privileged 1/1 Running 0 6s parisi@cloudshell:~$ kubectl exec -it nfs-poc-privileged -- configure-nfs.sh Stopping NFS common utilities: gssd idmapd statd. Starting NFS common utilities: statd idmapd gssd. Stopping NFS common utilities: gssd idmapd statd. Starting NFS common utilities: statd idmapd gssd. Stopping NFS common utilities: gssd idmapd statd. Starting NFS common utilities: statd idmapd gssd. /nfs is not mounted. Mounting /nfs... Mount success! parisi@cloudshell:~$ kubectl create -f unpriv-pod.yaml pod/nfs-poc-unprivileged created parisi@cloudshell:~$ kubectl get pods NAME READY STATUS RESTARTS AGE nfs-poc-privileged 1/1 Running 0 66s nfs-poc-unprivileged 1/1 Running 0 3s parisi@cloudshell:~$ kubectl exec -it nfs-poc-unprivileged -- bash root@nfs-poc-unprivileged:/# id parisi uid=1019(parisi) gid=1020(parisigroup) groups=1020(parisigroup),513(domain users) root@nfs-poc-unprivileged:/# ksu parisi -n parisi Changing uid to parisi (1019) bash: /home/parisi/.bashrc: Key has expired parisi@nfs-poc-unprivileged:/$ kinit Password for parisi@CVSDEMO.LOCAL: parisi@nfs-poc-unprivileged:/$ klist Ticket cache: FILE:/tmp/krb5cc_1019.hnXlZKVV Default principal: parisi@CVSDEMO.LOCAL Valid starting Expires Service principal 05/13/22 12:40:32 05/13/22 22:40:32 krbtgt/CVSDEMO.LOCAL@CVSDEMO.LOCAL renew until 05/20/22 12:40:29 parisi@nfs-poc-unprivileged:/$ cd /home parisi@nfs-poc-unprivileged:/home$ klist Ticket cache: FILE:/tmp/krb5cc_1019.hnXlZKVV Default principal: parisi@CVSDEMO.LOCAL Valid starting Expires Service principal 05/13/22 12:40:32 05/13/22 22:40:32 krbtgt/CVSDEMO.LOCAL@CVSDEMO.LOCAL renew until 05/20/22 12:40:29 05/13/22 12:40:40 05/13/22 22:40:32 nfs/nfsserver.cvsdemo.local@CVSDEMO.LOCAL renew until 05/20/22 12:40:29
You can find the GitHub repository here:
Here’s a little more information on what all was needed to make this work properly.
This was needed to ensure all the NFS services could start properly. This is what a listing of that path looks like:
# ls /run/rpc_pipefs cache gssd lockd mount nfs nfsd nfsd4_cb portmap statd
We did this by exposing the path via a VOLUME command in the container file.
# Attach rpc_pipefs RUN mkdir -p /run/rpc_pipefs VOLUME /run/rpc_pipefs
It’s also in the privileged-pod.yaml file:
- mountPath: /run/rpc_pipefs name: rpc-pipefs mountPropagation: Bidirectional
An improved configure-nfs.sh script
The previous NFS Kerberos container used a basic script that blindly restarted services on startup. While that works, it isn’t necessary. The new script will check if something is started and then start it. If it’s already started, we’ll just report back that it’s started. And, it also mounts the NFS Kerberos mount in the privileged container. This can be done two ways:
- Starting the container with bash (the script automatically runs)
- Starting the container with configure-nfs.sh (the script does all the necessary stuff without needing to enter the container)
There’s likely more improvement possible here, but you can find the script here.
Shared /tmp directory
By default, Kerberos will create krb5cc files when you kinit and place them in /tmp. With two different containers, that /tmp path needed to be shared to allow kinit/Kerberos access to work independently in the unprivileged container. Without this path, Kerberos only worked if you created the Kerberos ticket in the privileged container first. Exposing /tmp allows kinit to work in the unprivileged container without needing to enter the privileged container at all. You can also configure different paths for the krb5cc files in krb5.conf if you choose (including an NFS mount). More details on that cache here:
Those were the main changes made and everything seems to be humming right along. If you have questions, feel free to leave them in the comments!