Getting ONTAP NFS Kerberos to Work with FreeIPA

Image result for hamburglar

Obviously, with the social distancing/lockdowns happening, I have had more time to write up blogs on things. So, here’s another one. If you have suggestions for topics, let me know and I’ll see about writing them up.

Before we get to the meat of what the title is (or scroll down to the bottom if you want the quick and dirty steps), let’s recap NFS security and Kerberos…

NFS is a protocol that allows multiple clients communicate to a single storage endpoint as a way to share data across a network. Because data is transmitted over a network, it’s important to be able to secure that data, both at-rest and in-flight.

At-rest Security in NFS

At-rest security refers to security applied to data residing on the storage system, as well as the interaction between NFS client and NFS server to negotiate things like NFS version, identity, etc.

At-rest security for NFS includes:

  • Export policies and rules
  • Permissions/ACLs
  • User/group owners
  • Name ID strings (NFSv4.0 and later)

However, when data passes over the wire, packet sniffers are able to see identifying information like IP addresses, users, groups, permissions, file names, paths, etc. All it would take is someone being able to see this information to easily create duplicate set ups to “spoof” users and groups and gain access to data. At-rest security mostly protects you from threats that don’t have this knowledge or expertise. To prevent bad actors from getting information from the transmission of data, you’d need to set up in-flight security.

In-flight Security in NFS

In-flight security refers to securing the actual data packets in transit.

For NFS, this can be done in a few ways:

For ONTAP, tunneling NFS over SSH or stunnel isn’t supported, but you can use NFS Kerberos.

For a deeper look at NFS Kerberos in ONTAP, see:

NFS Kerberos in ONTAP Primer

Kerberos Security Flavors

NFS Kerberos has 3 methods that can be used to secure things. Each provides an additional level of security, but also adds extra performance overhead.

Keep in mind that NFSv3 *can* use Kerberos, but only the NFS portion of the protocol will use it. Ancillary protocols like mountd, portmapper, NLM, etc. will still be unencrypted. The most secure version of NFS available is NFS v4.x, which combines all the NFS ancillary protocols into a single port and compound NFS calls, which can all be Kerberized.

krb5 – Encrypts the authentication only

Remember when I said if a bad actor stole information about the client, user, etc. that they could spoof that user to get access? Well, with krb5, that becomes harder to do. You can have all the information about the NFS export, but to gain access to a Kerberized mount, you’d need to have a valid username and password to get a Kerberos TGT, which would then be used to get the NFS service ticket. In addition, your client would also need to have a Kerberos ticket and keytab to gain access, so unless your attacker has KDC admin rights, they won’t be able to access the mount.

krb5i – Authentication with data checksums/integrity checking

Krb5 secures the initial authentication for NFS, but the actual NFS data packets are still transmitted in clear text across the wire. If a bad actor were to plug in and sniff those data packets in flight, they could see everything. In addition, bad actors could also act as a “man in the middle” to intercept packets and then transmit their own data if they chose. Krb5i prevents this by using checksums on the client and server to verify that all the data arriving and leaving is coming from a trusted source. The data is still visible, but it can’t be interfered with.

krb5p – Authentication/integrity checking/end to end encryption (privacy)

For the ultimate in-flight security hammer for NFS, you would use krb5p. This combines what krb5i does with in-flight encryption of all NFS packets. That means all NFS packets will be encrypted with the enctype specified in the Kerberos configuration. This can add considerable overhead for NFS performance, but will also prevent NFS packets from being sniffed easily.

How to set up NFS Kerberos in ONTAP

Setting up Kerberos in ONTAP requires a few things:

  • A Key Distribution Center (Microsoft AD, MIT Kerberos, FreeIPA, RedHat IDM)
  • DNS server or static host entries
  • Optional (but recommended): LDAP server for UNIX identity management
  • Configuring NFS clients and ONTAP Service Principal Names

The KDC is the central hub for Kerberos operations and is responsible for handing out Kerberos tickets to clients, users and services for authentication in a Kerberos realm.

DNS is used to resolve IP addresses to host names, which are then passed to the KDC as SPN requests. For example, if client tries to get a Kerberos ticket, a DNS reverse lookup will be done to find the hostname FQDN “” Then a SPN request for host/ is made to the KDC. If the SPN exists and matches the request, then the Kerberos request moves on to the next steps. If it doesn’t exist, the request fails.

LDAP is used to centralize the UNIX identities for users and groups to ensure clients and servers have the same information all the time, without manual intervention. This is less important to Kerberos and more important to NFSv4.x and NFS permission resolution.

NFS client configuration for Kerberos on newer clients is fairly straightforward; you configure DNS (so it can find the Kerberos realm name and client hostname) and then simply use one of the automated tools to “join” the Kerberos realm. This automatically creates the service principal and transfers the keytab files. Where it gets tricky is if you have to do a manual Kerberos configuration. TR-4073 and TR-4616 can be of some guidance there, as well as a bevy of articles across the web.

NFS server/ONTAP configuration for Kerberos is relatively simple; you configure DNS and the Kerberos realm and then you enable Kerberos on a network interface. When you do that, ONTAP interacts with the KDC you specified in the Kerberos realm and automates the principal creation – provided the KDC is using standard Kerberos interaction, such as Microsoft Active Directory or kadmin for Linux.

Now we’re getting to the part the title hinted about… FreeIPA Kerberos setup.

Why do I need a blog on FreeIPA Kerberos with ONTAP? You just said it was easy!

I did say it was easy – if the KDC is using standard kadmin. However, FreeIPA happens to use a wrapper over kadmin for KDC management and discourages the use of kadmin for management of service principals. In fact, by default, they lock things down pretty tight.

In FreeIPA, there is a GUI to make things simpler to use. There are also “ipa” commands to interact via the CLI, such as ipa user-add or ipa service-add. In most Linux KDCs, there was kadmin to manage things. In fact, there are *two* kadmins. One is for local management (kadmin.local) and the other is for remote management (kadmin). For more info on the differences, see:

The remote kadmin is what ONTAP interacts with for Linux KDCs. When you use the automated ONTAP method to add the NFS service principal, the following happens:

  • A prompt for a username and password to issue a kinit to the KDC to gain access to kadmin
  • get_principal is run via remote kadmin to see if the SPN exists
    • If the SPN doesn’t exist, create_principal is run via remote kadmin
    • If the SPN does exist, modify_principal is run via remote kadmin

All of these kadmin commands require the proper privileges to run successfully. In FreeIPA, remote kadmin is locked down by default to even run get_principal.

For example:

# ipa user-add krbadmin

Added user "krbadmin"
User login: krbadmin
First name: Kerberos
Last name: Admin
Full name: Kerberos Admin
Display name: Kerberos Admin
Initials: KA
Home directory: /home/krbadmin
GECOS: Kerberos Admin
Login shell: /bin/sh
Principal name: krbadmin@CENTOS-LDAP.LOCAL
Principal alias: krbadmin@CENTOS-LDAP.LOCAL
Email address: krbadmin@centos-ldap.local
UID: 1971600007
GID: 1971600007
Password: False
Member of groups: ipausers
Kerberos keys available: False

# kadmin -p krbadmin
Authenticating as principal krbadmin with password.
Password for krbadmin@CENTOS-LDAP.LOCAL:
kadmin: get_principal nfs/
get_principal: Operation requires ``get'' privilege while retrieving "nfs/".

The ONTAP error we’d see is less clear, however. The error suggests that the SPN *already exists*.

Error: NFS Kerberos bind SPN procedure failed
  [  0 ms] Creating account in Unix KDC
  [    30] Successfully connected to ip 10.x.x.x, port 749
           using TCP
**[    40] FAILURE: Unexpected state: Error 1142 at
**         file:src/utils/secd_kadmin_utils.cpp
**         func:createVifKrbAccountUsingKadmin line:227
**[    40] FAILURE: spn already exists. Failed to reuse spn
**         'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
**         admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
**         Unknown code 0
  [    41] Uncaptured failure while creating account

Obviously, the error should refer to some permissions issue. When we see this error in ONTAP,  we can verify on the KDC what is happening via the /var/log/kadmind.log file. In this case, we see “unauthorized request.”

Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Request: kadm5_init, kadmin/admin@CENTOS-LDAP.LOCAL, success, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x, vers=2, flavor=6
Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Unauthorized request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

To give permissions to users to use remote kadmin, you have to modify the /var/kerberos/krb5kdc/kadm5.acl file.

In my setup, this is how I configured the ACL:

# cat /var/kerberos/krb5kdc/kadm5.acl

Then you restart IPA services:

# service ipa restart
Redirecting to /bin/systemctl restart ipa.service

Now, my user can query the principal:

kadmin: get_principal nfs/
get_principal: Principal does not exist while retrieving "nfs/".

However, ONTAP gets a new (less descriptive) error when trying to interact with the KDC:

Error: NFS Kerberos bind SPN procedure failed
[ 1 ms] Creating account in Unix KDC
[ 29] Successfully connected to ip x.x.x.x, port 749
using TCP
**[ 45] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:createVifKrbAccountUsingKadmin line:223
**[ 45] FAILURE: Failed to create spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
** Invalid argument
[ 45] Uncaptured failure while creating account

The /var/log/kadmind.log is also fairly useless here:

Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Principal does not exist, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x
Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_create_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid argument, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

ONTAP is just reporting what kadmin says!

Our clue comes from when we try to manually create the SPN using kadmin:

kadmin: add_principal -randkey nfs/
WARNING: no policy specified for nfs/; defaulting to no policy
add_principal: Kerberos database constraints violated while creating "nfs/".

Same goes for modify_principal:

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
(vserver nfs kerberos interface enable)

Username: ontap


Warning: An account that matches the given service principal name already exists. Re-using this account deletes and re-creates the account if it is not shared by the LIFs in Vserver "NFS". This
invalidates any existing Kerberos service tickets and keys for this account. Do you want to re-use this account? {y|n}: y

Error: NFS Kerberos bind SPN procedure failed
[ 0 ms] Creating account in Unix KDC
[ 39] Successfully connected to ip x.x.x.x, port 749
using TCP
[ 45] Re-using the account for
**[ 53] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:randomizePasswordUsingKadmin line:287
**[ 53] FAILURE: Failed to randomize password for spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'ontap@CENTOS-LDAP.LOCAL'
[ 54] Uncaptured failure while creating account

Mar 24 11:14:54 centos8-ipa.centos-ldap.local kadmind[17208](Notice): Request: kadm5_randkey_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid key/salt tuples, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

And delete_principal:

::*> kerberos interface disable -vserver NFS -lif ipa-krb
(vserver nfs kerberos interface disable)

Username: ontap


Warning: This command deletes the service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" from the machine account on the KDC. Do you want to continue? {y|n}: y

Error: command failed: Failed to disable NFS Kerberos on LIF "ipa-krb". Failed to delete the account associated with the Kerberos service principal name. Reason: cifs smb kadmin error.

So that means, even though we gave full rights to that user in the kadm5.acl file, we can’t create principals in FreeIPA using kadmin. They want to funnel you to use the IPA tools – likely for a good reason. There’s probably a bunch of other automated steps that take place with those tools, so this is in the interest of simplicity. and security.

How do we get past this?

There are two options here.

  1. Open up the permissions on FreeIPA to use kadmin to create/add/modify/delete principals. (I haven’t figured out how to do this yet)
  2. Manually create the SPN and keytab file and then use ONTAP to transfer the keytab via URI

I went with option 2.

The TL;DR steps, as provided by Alexander Bokovoy in the comments:

- kinit admin
- ipa host-add demo-ipa.centos-ldap.local
- ipa service-add nfs/demo-ipa.centos-ldap.local
- ipa-getkeytab -p nfs/demo-ipa.centos-ldap.local -k ./nfs.keytab -e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

Remember that ONTAP only supports the following enctypes for NFS Kerberos:

  • AES-128 and AES-256
  • DES and 3DES

Then, you copy that file to your HTTP or FTP server. The address to the file will be used in the ONTAP CLI command.

Note: If you’re using IIS, you may have to allow .keytab as a MIME type.

For example:


Once the file is web-accessible, you run the kerberos interface enable command and use the -keytab-uri option to upload the keytab. Unsupported enctypes will get discarded.

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL -keytab-uri http://web-server/files/ipakrb-ontap.keytab
(vserver nfs kerberos interface enable)

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

Warning: Keys for encryption types "des-cbc-crc,des3-cbc-sha1,aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96" are required for Vserver "NFS" but found keys only for encryption types
"aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96". Keys for encryption types "des-cbc-crc,des3-cbc-sha1" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" are
missing. Available keys will be imported. Do you want to continue? {y|n}: y

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

You’ll also want to ensure you have a valid UNIX user or name mapping rule for the krb-spn name mapping for the NFS clients. That’s covered in TR-4616 in detail.

Once that’s all done, NFS Kerberos mounts should work fine!

[root@centos8-1 sysconfig]# mount -o nfsvers=4,minorversion=1,sec=krb5 x.x.x.x:/kerberos /mnt
[root@centos8-1 sysconfig]# mount | grep krb
x.x.x.x:/kerberos on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.y,local_lock=none,addr=x.x.x.x)
# su ipa-user
sh-4.4$ cd /mnt
sh: cd: /mnt: Permission denied
sh-4.4$ kinit
Password for ipa-user@CENTOS-LDAP.LOCAL:
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ cd /mnt
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
03/23/2020 14:40:16 03/24/2020 14:39:27 nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ touch krb5file
sh-4.4$ ls -la
total 4
drwxrwxrwx. 2 root root 4096 Mar 23 14:40 .
dr-xr-xr-x. 17 root root 224 Jan 21 10:03 ..
-rw-------. 1 ipa-user admins 0 Mar 23 14:40 krb5file
-rw-r--r--. 1 root root 0 Mar 23 14:15 v3nokrb
-rw-r--r--. 1 root root 0 Mar 23 14:16 v4nokrb
-rw-------. 1 ipa-user admins 0 Mar 23 14:22 v4nokrb-ipa

I opened a bug to make this easier in ONTAP with FreeIPA (1308667), so if you run into this, open a case and have it attached to the bug.

If you have questions, suggestions or get stuck (or if I got something wrong), comment below!

Encrypt your NFS packets end to end with krb5p and ONTAP 9.2!

NFS has always had a running joke about security, with a play on the acronym stating that NFS was “Not For Security.”

With NFSv3 and prior, there was certainly truth to that, especially when NFS was mounted without Kerberos. But even using Kerberos in NFSv3 wasn’t necessarily secure, as it only was applied to the NFS packets and not the extraneous services like NLM, NSM, mountd, etc.

NFSv4.x improved NFS security greatly by implementing a single port, ACLs, ID domain names and more tightly integrated support for Kerberos, among other improvements. However, simple krb5 authentication by itself only encrypts the initial mounts and not the NFS packets themselves.

That’s where stronger Kerberos modes like krb5i and krb5p come into play. From the RedHat man pages:

sec=krb5 uses Kerberos V5 instead of local UNIX UIDs and GIDs to authenticate users.

sec=krb5i uses Kerberos V5 for user authentication and performs integrity checking of NFS operations using secure checksums to prevent data tampering.

sec=krb5p uses Kerberos V5 for user authentication, integrity checking, and encrypts NFS traffic to prevent traffic sniffing. This is the most secure setting, but it also involves the most performance overhead.

krb5p = privacy

The p in krb5p stands for “privacy,” and it does that by way of Kerberos encryption of the NFS conversation end-to-end, via the specified encryption strength. The strongest you can currently use is AES-256. ONTAP 9.0 and later supports krb5p and AES-256 encryption. Krb5p is similar to SMB3 encryption/signing and sealing in its functionality.

Krb5p is also similar to SMB3 encryption in its performance impact; doing encryption of thousands of packets is expensive and can create CPU bottlenecks, unless…

AES-NI Offloading

AES-NI offloading is a feature available on specific Intel CPUs that allow encryption processing to use hardware acceleration instructions to offload processing for encryption. This allows the encryption to be done separately to alleviate performance bottlenecks.

From Intel’s site:

Intel® AES New Instructions (Intel® AES NI) is a new encryption instruction set that improves on the Advanced Encryption Standard (AES) algorithm and accelerates the encryption of data in the Intel® Xeon® processor family and the Intel® Core™ processor family.

Comprised of seven new instructions, Intel® AES-NI gives your IT environment faster, more affordable data protection and greater security; making pervasive encryption feasible in areas where previously it was not.

ONTAP 9.1 provided support for AES-NI offloading for SMB3 encryption, which greatly improved performance. But krb5p offloading was only added as of ONTAP 9.2. If you plan on using the end-to-end encryption functionality in NFS with krb5p, use ONTAP 9.2 or later. For more information on what other features are in ONTAP 9.2, see the following post:

ONTAP 9.2RC1 is available!

Krb5p performance in ONTAP 9.0 vs. ONTAP 9.2

Krb5p support was added in ONTAP 9.0, but the performance was pretty awful, due to the lack of AES-NI support.

Here are some graphs using SIO with different flavors of Kerberos and AUTH_SYS in ONTAP 9.0. (All using NFSv4.1)

In ONTAP 9.0, krb5p wasn’t ever able to achieve above 12k IOPS for 4k reads in these SIO tests, and what it was able to achieve, it did it at some pretty severe latency. Krb5i did a little better, but krb5 and auth_sys performed way better.

Test environment was:

  • FAS8080 (AFF numbers coming soon)
  • 12 RHEL 6.7 clients

4K sequential reads in ONTAP 9.0:


Writes are even worse for krb5p in ONTAP 9.0 – we didn’t even get to 10k.

4K sequential writes in ONTAP 9.0:


For 8K sequential reads in ONTAP 9.0, latency is about the same. Fewer ops, but that’s because we’re doing the same amount of work in bigger I/O chunks.

8K sequential reads in ONTAP 9.0:


8K sequential writes in ONTAP 9.0:


NOTE: ONTAP 9.1 was not tested, but I’d expect similar performance, as we don’t do AES-NI offloading for NFS in that release.

ONTAP 9.2 Kerberos 5p Performance – Vastly improved

Now, let’s compare those same tests to ONTAP 9.2 with the AES-NI offloading and other performance enhancements. In the graphs below, there are a few things to point out.

  • Much more predictable performance for krb5i and krb5p as IOPS increase
  • Lower latency in 9.2 at high IOPS for krb5 than in 9.0
  • No real peak IOPS for krb5i/krb5p; these security flavors are able to keep up with sys and krb5 for sheer maximum IOPS
  • Sub millisecond latency for NFS at high IOPS (~50k) in most workloads, regardless of the security flavor
  • AES-NI offloading and NFS performance improvements in ONTAP 9.2 are pretty substantial

4K Sequential Reads in ONTAP 9.2:


4K Sequential Writes in ONTAP 9.2:


8K sequential reads in ONTAP 9.2:


8K sequential writes in ONTAP 9.2:



With ONTAP 9.2, you can now get enterprise class security with Kerberos 5p along with performance that doesn’t kill your workloads. If you’re doing NFS with any flavor of Kerberos, it makes a ton of sense to upgrade to ONTAP 9.2 to receive the performance benefits from AES-NI offloading. Keep in mind that upgrading ONTAP is non-disruptive to NFSv3, as it’s stateless, but will be slightly disruptive to CIFS/SMB and NFSv4.x workloads, due to the statefulness of the protocols.