It’s a Kerberos Khristmas!

Recently, I was working on a project where I was creating a POC of an Ubuntu container that could authenticate to LDAP and mount NFS Kerberos mounts without any interaction that would eventually be used in a Kubernetes environment. It was an improvement on the container image I created a while back in “Securing NFS mounts in a Docker container,” and I’ll write more about it in another post later.

But, since it’s almost Christmas, I wanted to deliver a few gifts I discovered during this process that can help with other deployments.

KSU

No, I didn’t discover Kansas State University.

What I *did* discover was a way to “su” as a user while also running kinit at the same time. The ksu utility allows you to do that with the following:

# ksu username -n username

When you run that, you “su” as the UID and kinit kicks off:

root@cfeac39a405f:/# ksu student1 -n student1
WARNING: Your password may be exposed if you enter it here and are logged
in remotely using an unsecure (non-encrypted) channel.
Kerberos password for student1@NTAP.LOCAL: :
Changing uid to student1 (1301)

And this is the ticket (NFS service ticket came with the autofs mount of the homedir):

student1@cfeac39a405f:/$ klist
Ticket cache: FILE:/tmp/krb5cc_1301.kufVdH57
Default principal: student1@NTAP.LOCAL
Valid starting Expires Service principal
12/23/21 12:04:15 12/23/21 13:04:15 krbtgt/NTAP.LOCAL@NTAP.LOCAL
renew until 12/30/21 12:04:15
12/23/21 12:04:16 12/23/21 13:04:15 nfs/demo.ntap.local@NTAP.LOCAL
renew until 12/30/21 12:04:15

student1@cfeac39a405f:/$ mount | grep home
auto.home on /home type autofs (rw,relatime,fd=18,pgrp=118,timeout=50,minproto=5,maxproto=5,indirect,pipe_ino=1359512)

Pretty cool, but not as cool as the next thing I discovered…

msktutil

Getting Kerberos keytabs on Linux clients has traditionally been a pain – especially when using Active Directory as the KDC. Commands like “net ads” and “realm join” have made this simpler, but they usually require admin interaction and with containers, you can’t always do that. Nor would you really want to – who wants to join every container you create to the domain?

For Kerberos mounts, a machine account keytab is used for the initial mount to the ONTAP NFS server and the SPN you create will map into ONTAP and try to authenticate to a valid UNIX user that the ONTAP SVM has to know about. In AD KDCs, when you create a keytab, you have to map it to a user, which then populate the userPrincipalName field, which is then passed to ONTAP as the incoming SPN.

With CVS/ANF instances of ONTAP, you are at the mercy of whatever the instance has configured, so you need to either configure the SPN to MACHINE$@DOMAIN.COM or root/fqdn.domain.com@DOMAIN.COM for things to work without any other configuration.

In AD, there’s a utility called “ktpass,” which works fine, but then you have to worry about transferring the keytab file to the Linux client every time you change it and if you want to rotate keytabs on a regular basis, this gets to be cumbersome.

What I like to do with containers is have them all share a common keytab for their LDAP bind with SSSD and the NFS mount authentication. Regular users won’t be able to access the mount without using kinit anyway, and most containers don’t require a huge level of security on the container side. Kerberos is usually just there to encrypt the traffic in flight.

Since we need a keytab for containers to share located in a common location and we don’t really want to share the same keytab as the host, we can’t necessarily join the domain with “realm.” And since we’re not AD domain admins with regular access to the KDCs, ktpass is hard to manage as well.

The msktutil tool allows interaction with the KDC for keytab creation on the local Linux machine, as well as keytab updates if you want to periodically refresh the keytabs (which you should, for better overall security).

All you need to do on the Linux client is kinit as a user with permissions on the KDC to create objects. Then, when you run the “msktutil create” command, it will create a machine account in AD and a keytab based on your desired parameters. This sample command creates a keytab with AES-256 encryption and a SPN/UPN of root/fqdn:

# msktutil create --verbose --computer-name UBUNTU-CONTAINER -k ubuntu-container.keytab --enable --enctypes 0x10 --service root/ubuntu-container.ntap.local --upn root/ubuntu-container.ntap.local

This is the resulting keytab file:

# klist -kte ubuntu-container.keytab
Keytab name: FILE:ubuntu-container.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 12/21/2021 22:33:34 UBUNTU-CONTAINER$@NTAP.LOCAL (aes256-cts-hmac-sha1-96)
   1 12/21/2021 22:33:34 root/ubuntu-container.ntap.local@NTAP.LOCAL (aes256-cts-hmac-sha1-96)
   1 12/21/2021 22:33:34 host/centos83-perf2@NTAP.LOCAL (aes256-cts-hmac-sha1-96)

When that attempts to authenticate to the ONTAP SVM, root/ubuntu-container maps to root and mounts work fine.

If I wanted to update that keytab file, I can run this:

# msktutil update --verbose --computer-name UBUNTU-CONTAINER -k ubuntu-container.keytab --enable --enctypes 0x10 --service root/ubuntu-container.ntap.local --upn root/ubuntu-container.ntap.local

When I do that, the keytab updates and the kvno (Key version number) changes:

# klist -kte ubuntu-container.keytab
Keytab name: FILE:ubuntu-container.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   1 12/21/2021 22:33:34 UBUNTU-CONTAINER$@NTAP.LOCAL (aes256-cts-hmac-sha1-96)
   1 12/21/2021 22:33:34 root/ubuntu-container.ntap.local@NTAP.LOCAL (aes256-cts-hmac-sha1-96)
   1 12/21/2021 22:33:34 host/centos83-perf2@NTAP.LOCAL (aes256-cts-hmac-sha1-96)
   2 12/23/2021 12:23:59 UBUNTU-CONTAINER$@NTAP.LOCAL (aes256-cts-hmac-sha1-96)
   2 12/23/2021 12:23:59 root/ubuntu-container.ntap.local@NTAP.LOCAL (aes256-cts-hmac-sha1-96)
   2 12/23/2021 12:23:59 host/centos83-perf2@NTAP.LOCAL (aes256-cts-hmac-sha1-96)

If I don’t want extra entries, I can delete the old keytab and re-create it. Once you rotate the keytab, you may have to re-build the container to ensure the keytab file is updated in the container.

Restart services when a container starts

One challenge I faced when setting these up was that the LDAP and NFS services didn’t always work properly when you first start the container. So I found myself having to manually restart the services. When you’re running a container, the idea is that you shouldn’t have to babysit them. So, I found that creating a simple shell script to restart services, like this:

#!#!/bin/sh
sudo service dbus start
sudo /sbin/rpcbind
sudo service nfs-common restart
sudo /usr/sbin/rpc.gssd
sudo /usr/sbin/rpc.svcgssd
sudo service sssd start
sudo service autofs start
sudo service sssd restart

That script gets copied during the container run with these lines in the dockerfile:

# Script to start services
COPY configure-nfs-ubuntu.sh /usr/local/bin/configure-nfs-ubuntu.sh
RUN chmod +x /usr/local/bin/configure-nfs-ubuntu.sh

And then adding this line to the end of .bashrc did the trick:

sh /usr/local/bin/configure-nfs-ubuntu.sh

And this is how it looks when the container starts:

# docker exec -it containername bash
 * system message bus already started; not starting.
rpcbind: another rpcbind is already running. Aborting
 * Stopping NFS common utilities                                                                                                                      [ OK ]
 * Starting NFS common utilities                                                                                                                      [ OK ]
 * Starting automount... 

That’s all I have for now. Stay tuned for more on this topic, including updates to the “This is the Way” series detailing my first installation of Kubernetes. Feel free to comment below!

Advertisement