Mounting ONTAP CIFS/SMB shares with Linux – Guidelines and tips

Every now and then, the question of mounting CIFS/SMB shares in ONTAP using Linux clients comes up. Many people might consider this configuration to be a crime against nature.

If you’re asking yourself “why would someone want to do this,” there are a few reasons.

• Lock handling – SMB locks use oplocks. NFS locks use shared locks or delegations, depending on the NFS version and options being used. Generally this is fine, but in some cases, the style of lock might create complications with some applications. For example, Kafka on NFS has some issues with how NFS handles deletions when a file is locked:
  https://blog.usejournal.com/kafka-on-kubernetes-a-good-fit-95251da55837?gi=91867e828261
• Legacy copiers/scanners – Many companies will use copiers/scanners to create image files that redirect to a NAS share in their environment. Many of those copiers use old SMB clients and are no longer under support, so they can’t be upgrades. And many of those scanners/copiers are expensive, so people aren’t interested in buying newer ones just to get newer SMB versions.
• “How we’ve always done it” – This is a common reason that doesn’t always have technical logic associated with it, but you still have to roll with it.

There are other reasons out there as well, I’m sure. But this post will attempt to group the considerations, support statements and issues you might see in case someone else has that question.

Mounting SMB shares in ONTAP – General Guidelines

In 7-Mode ONTAP, mounting SMB shares with clients like AIX had no issues. This was because 7-Mode supported a lot of old SMB technology concepts that newer versions of ONTAP have deprecated. Even though SMB shares on Linux worked in 7-Mode, there was never an official supported configuration for it.

NetApp ONTAP has traditionally only supported Apple OS and Windows OS for CIFS/SMB workloads.

You can find the supported clients in the Interoperability Matrix here:

https://mysupport.netapp.com/matrix/

With newer versions of ONTAP, you *can* get SMB/CIFS working properly, provided you take the following into account:

• CIFS/SMB clients should support UNICODE for CIFS/SMB communication. Clustered ONTAP doesn’t support non-UNICODE in any release and never will. Newer CIFS/SMB clients should support UNICODE. AIX is a good example of an older Linux client that has some challenges using SMB because it still uses non-UNICODE in some client installs.
• CIFS/SMB versions mounting should reflect the supported SMB versions in ONTAP. ONTAP has deprecated SMB1, so you’d need to mount using SMB2 or SMB3.
• CIFS/SMB enabled features in ONTAP should reflect the feature support for the CIFS/SMB Linux clients. For example, my Centos7 Linux Samba client doesn’t support large MTU, SMB3 encryption, multichannel, etc. That can cause failures in the mounts (for an example, see below).
• If using Kerberos for the CIFS/SMB mounts, be sure the CIFS/SMB ONTAP server has a valid SPN for the CIFS service. The name should match the mount name. For example, if your CIFS/SMB server is named “CIFS” then the SPN would be cifs/cifs.domain.com.

Common Issues when Mounting CIFS/SMB

One of the main issues people run into with SMB mounts in Linux (provided they follow the guidelines above when using ONTAP) is using the wrong command syntax. For example, when you specify the share path, you use either \\\\name\\share (extra slashes to tell Linux that the \ isn’t an operand) or you use //name/share.

The following NetApp KB article does a good job of giving command examples for the right way to mount SMB:

How to access CIFS from Linux machine using SAMBA

Usually the command will tell you if the wrong option is being used, but sometimes the errors are less than helpful. These are a few errors I ran into.

Incorrect Mount Option

This was pretty self-explanatory. First, I didn’t have the cifs-utils installed. Second, I used the wrong command syntax.

Mount point does not exist

Again, self explanatory. The directory you’re trying to mount to needs to exist.

No such file or directory (when mounting)

This means you specified the wrong export path on the NFS server. Check your junction-paths and try again.

Host is down

This one was a tricky error, as it suggests that the server was not up. In some cases, that may truly be the issue. But it wasn’t in my case.

# mount -t cifs -o user=cifsuser \\\\DEMO\\nas /mnt/nas
Password for cifsuser@\DEMO\nas: **********
mount error(112): Host is down

But, in reality, the issue was that I didn’t specify the SMB version and it tried to use SMBv1.0 by default.

Specifying the SMB version (-o vers=3.0) got past that issue.

Required key not available

This is a Kerberos specific error. In my case, the cifs/servername.domain.com SPN didn’t exist for the hostname I used in the UNC path. You can see that in a packet capture.

Permission denied

This error is fairly useless in both Windows and Linux in a lot of cases – mainly because it can mean a variety of things. Sometimes, it really is an access issue (such as share or file-level permissions). But in my testing, I also ran into this issue when I had unsupported SMB features enabled on my CIFS server in ONTAP.

# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas
Password for administrator@//DEMO/nas: **********
mount error(13): Permission denied

In a packet trace, I could see the client telling me what it supported:

But the reply didn’t really mention the issue. So I made sure to disable the following CIFS/SMB features:

• SMB3 encryption (cifs security modify)
• Large MTU and SMB Multichannel (cifs options modify)

Once I did that, I was able to mount.

[root@centos7 /]# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas
Password for administrator@//DEMO/nas: **********
[root@centos7 /]# touch /mnt/nas/smbfile
[root@centos7 /]# ls -la /mnt/nas
total 1
drwxr-xr-x 2 root root 0 Mar 27 14:36 .
drwxr-xr-x. 9 root root 97 Mar 27 10:17 ..
-rwxr-xr-x 1 root root 0 Mar 27 14:36 smbfile

For Kerberos, you would just specify sec=krb5.

# mount -t cifs -o sec=krb5,vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas

Stale file handle errors when crossing junction paths

In some cases, if you mount volumes to volumes (via junction-path), newer Linux clients may report different inode numbers for the junctioned volume when using a command like “ls -i” than when using “ls -id”. This can create potential access problems from clients, such as “stale file handle” errors.

For example, this Centos 8.3 client shows different inode numbers for the same volume:

# ls -i
8107029540347314273 mountedvolume
# ls -id /mnt/cifs/mountedvolume/
2744251190362505280 /mnt/cifs/mountedvolume/

The way around this issue is to mount with the mount option “actimeo=0.” Newer SMB clients seem to cache inode information and create potential conflicts. Disabling actimeo prevents that caching from taking place. Since this is a caching option, you might see slower overall performance, but it will fix the stale file handle issue. Alternately, you may want to mount with the “noserverino” option instead, which doesn’t try to use the SMB server’s inodes.

Inode Numbers

When Unix Extensions are enabled, we use the actual inode number provided by the server in response to the POSIX calls as an inode number.

When Unix Extensions are disabled and "serverino" mount option is enabled there is no way to get the server inode number. The client typically maps the server-assigned "UniqueID" onto an inode number.

Note that the UniqueID is a different value from the server inode number. The UniqueID value is unique over the scope of the entire server and is often greater than 2 power 32. This value often makes programs that are not compiled with LFS (Large File Support), to trigger a glibc EOVERFLOW error as this won't fit in the target structure field. It is strongly recommended to compile your programs with LFS support (i.e. with -D_FILE_OFFSET_BITS=64) to prevent this problem. You can also use "noserverino" mount option to generate inode numbers smaller than 2 power 32 on the client. But you may not be able to detect hardlinks properly.

Once I set actimeo=0, this is how those inodes looked:

# ls -i
8107029540347314273 mountedvolume
# ls -id /mnt/cifs/mountedvolume/
8107029540347314273 /mnt/cifs/mountedvolume/

There you go! Hopefully this helped someone from going down a Google K-hole. Leave comments or suggestions below.

Behind the Scenes: Episode 234 – NetApp Keystone

Welcome to the Episode 234, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

This week on the podcast, we discuss NetApp Keystone with Sunitha Rao (Sunitha.rao@netapp.com, @sira_10) and Jean Banko (@bankonAI) and how it is simplifying cloud consumption.

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 234: NetApp Keystone – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Getting ONTAP NFS Kerberos to Work with FreeIPA

Obviously, with the social distancing/lockdowns happening, I have had more time to write up blogs on things. So, here’s another one. If you have suggestions for topics, let me know and I’ll see about writing them up.

Before we get to the meat of what the title is (or scroll down to the bottom if you want the quick and dirty steps), let’s recap NFS security and Kerberos…

NFS is a protocol that allows multiple clients communicate to a single storage endpoint as a way to share data across a network. Because data is transmitted over a network, it’s important to be able to secure that data, both at-rest and in-flight.

At-rest Security in NFS

At-rest security refers to security applied to data residing on the storage system, as well as the interaction between NFS client and NFS server to negotiate things like NFS version, identity, etc.

At-rest security for NFS includes:

• Export policies and rules
• Permissions/ACLs
• User/group owners
• Name ID strings (NFSv4.0 and later)

However, when data passes over the wire, packet sniffers are able to see identifying information like IP addresses, users, groups, permissions, file names, paths, etc. All it would take is someone being able to see this information to easily create duplicate set ups to “spoof” users and groups and gain access to data. At-rest security mostly protects you from threats that don’t have this knowledge or expertise. To prevent bad actors from getting information from the transmission of data, you’d need to set up in-flight security.

In-flight Security in NFS

In-flight security refers to securing the actual data packets in transit.

For NFS, this can be done in a few ways:

• NFS over SSH
• NFS over stunnel
• Network security like IPsec
• NFS Kerberos

For ONTAP, tunneling NFS over SSH or stunnel isn’t supported, but you can use NFS Kerberos.

For a deeper look at NFS Kerberos in ONTAP, see:

NFS Kerberos in ONTAP Primer

Kerberos Security Flavors

NFS Kerberos has 3 methods that can be used to secure things. Each provides an additional level of security, but also adds extra performance overhead.

Keep in mind that NFSv3 *can* use Kerberos, but only the NFS portion of the protocol will use it. Ancillary protocols like mountd, portmapper, NLM, etc. will still be unencrypted. The most secure version of NFS available is NFS v4.x, which combines all the NFS ancillary protocols into a single port and compound NFS calls, which can all be Kerberized.

krb5 – Encrypts the authentication only

Remember when I said if a bad actor stole information about the client, user, etc. that they could spoof that user to get access? Well, with krb5, that becomes harder to do. You can have all the information about the NFS export, but to gain access to a Kerberized mount, you’d need to have a valid username and password to get a Kerberos TGT, which would then be used to get the NFS service ticket. In addition, your client would also need to have a Kerberos ticket and keytab to gain access, so unless your attacker has KDC admin rights, they won’t be able to access the mount.

krb5i – Authentication with data checksums/integrity checking

Krb5 secures the initial authentication for NFS, but the actual NFS data packets are still transmitted in clear text across the wire. If a bad actor were to plug in and sniff those data packets in flight, they could see everything. In addition, bad actors could also act as a “man in the middle” to intercept packets and then transmit their own data if they chose. Krb5i prevents this by using checksums on the client and server to verify that all the data arriving and leaving is coming from a trusted source. The data is still visible, but it can’t be interfered with.

krb5p – Authentication/integrity checking/end to end encryption (privacy)

For the ultimate in-flight security hammer for NFS, you would use krb5p. This combines what krb5i does with in-flight encryption of all NFS packets. That means all NFS packets will be encrypted with the enctype specified in the Kerberos configuration. This can add considerable overhead for NFS performance, but will also prevent NFS packets from being sniffed easily.

How to set up NFS Kerberos in ONTAP

Setting up Kerberos in ONTAP requires a few things:

• A Key Distribution Center (Microsoft AD, MIT Kerberos, FreeIPA, RedHat IDM)
• DNS server or static host entries
• Optional (but recommended): LDAP server for UNIX identity management
• Configuring NFS clients and ONTAP Service Principal Names

The KDC is the central hub for Kerberos operations and is responsible for handing out Kerberos tickets to clients, users and services for authentication in a Kerberos realm.

DNS is used to resolve IP addresses to host names, which are then passed to the KDC as SPN requests. For example, if client 10.10.10.10 tries to get a Kerberos ticket, a DNS reverse lookup will be done to find the hostname FQDN “client1.domain.com” Then a SPN request for host/client1.domain.com@DOMAIN.COM is made to the KDC. If the SPN exists and matches the request, then the Kerberos request moves on to the next steps. If it doesn’t exist, the request fails.

LDAP is used to centralize the UNIX identities for users and groups to ensure clients and servers have the same information all the time, without manual intervention. This is less important to Kerberos and more important to NFSv4.x and NFS permission resolution.

NFS client configuration for Kerberos on newer clients is fairly straightforward; you configure DNS (so it can find the Kerberos realm name and client hostname) and then simply use one of the automated tools to “join” the Kerberos realm. This automatically creates the service principal and transfers the keytab files. Where it gets tricky is if you have to do a manual Kerberos configuration. TR-4073 and TR-4616 can be of some guidance there, as well as a bevy of articles across the web.

NFS server/ONTAP configuration for Kerberos is relatively simple; you configure DNS and the Kerberos realm and then you enable Kerberos on a network interface. When you do that, ONTAP interacts with the KDC you specified in the Kerberos realm and automates the principal creation – provided the KDC is using standard Kerberos interaction, such as Microsoft Active Directory or kadmin for Linux.

Now we’re getting to the part the title hinted about… FreeIPA Kerberos setup.

Why do I need a blog on FreeIPA Kerberos with ONTAP? You just said it was easy!

I did say it was easy – if the KDC is using standard kadmin. However, FreeIPA happens to use a wrapper over kadmin for KDC management and discourages the use of kadmin for management of service principals. In fact, by default, they lock things down pretty tight.

In FreeIPA, there is a GUI to make things simpler to use. There are also “ipa” commands to interact via the CLI, such as ipa user-add or ipa service-add. In most Linux KDCs, there was kadmin to manage things. In fact, there are *two* kadmins. One is for local management (kadmin.local) and the other is for remote management (kadmin). For more info on the differences, see:

https://docs.oracle.com/cd/E19683-01/816-0211/6m6nc66tj/index.html

The remote kadmin is what ONTAP interacts with for Linux KDCs. When you use the automated ONTAP method to add the NFS service principal, the following happens:

• A prompt for a username and password to issue a kinit to the KDC to gain access to kadmin
• get_principal is run via remote kadmin to see if the SPN exists
  • If the SPN doesn’t exist, create_principal is run via remote kadmin
  • If the SPN does exist, modify_principal is run via remote kadmin

All of these kadmin commands require the proper privileges to run successfully. In FreeIPA, remote kadmin is locked down by default to even run get_principal.

For example:

# ipa user-add krbadmin

---------------------
Added user "krbadmin"
---------------------
User login: krbadmin
First name: Kerberos
Last name: Admin
Full name: Kerberos Admin
Display name: Kerberos Admin
Initials: KA
Home directory: /home/krbadmin
GECOS: Kerberos Admin
Login shell: /bin/sh
Principal name: krbadmin@CENTOS-LDAP.LOCAL
Principal alias: krbadmin@CENTOS-LDAP.LOCAL
Email address: krbadmin@centos-ldap.local
UID: 1971600007
GID: 1971600007
Password: False
Member of groups: ipausers
Kerberos keys available: False

# kadmin -p krbadmin
Authenticating as principal krbadmin with password.
Password for krbadmin@CENTOS-LDAP.LOCAL:
kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Operation requires ``get'' privilege while retrieving "nfs/server.domain.com@DOMAIN.COM".

The ONTAP error we’d see is less clear, however. The error suggests that the SPN *already exists*.

Error: NFS Kerberos bind SPN procedure failed
  [  0 ms] Creating account in Unix KDC
  [    30] Successfully connected to ip 10.x.x.x, port 749
           using TCP
**[    40] FAILURE: Unexpected state: Error 1142 at
**         file:src/utils/secd_kadmin_utils.cpp
**         func:createVifKrbAccountUsingKadmin line:227
**[    40] FAILURE: spn already exists. Failed to reuse spn
**         'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
**         admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
**         Unknown code 0
  [    41] Uncaptured failure while creating account

Obviously, the error should refer to some permissions issue. When we see this error in ONTAP,  we can verify on the KDC what is happening via the /var/log/kadmind.log file. In this case, we see “unauthorized request.”

Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Request: kadm5_init, kadmin/admin@CENTOS-LDAP.LOCAL, success, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x, vers=2, flavor=6
Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Unauthorized request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

To give permissions to users to use remote kadmin, you have to modify the /var/kerberos/krb5kdc/kadm5.acl file.

In my setup, this is how I configured the ACL:

# cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@CENTOS-LDAP.LOCAL *
ontap@CENTOS-LDAP.LOCAL *
krbadmin@CENTOS-LDAP.LOCAL *

Then you restart IPA services:

# service ipa restart
Redirecting to /bin/systemctl restart ipa.service

Now, my user can query the principal:

kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Principal does not exist while retrieving "nfs/server.domain.com@DOMAIN.COM".

However, ONTAP gets a new (less descriptive) error when trying to interact with the KDC:

Error: NFS Kerberos bind SPN procedure failed
[ 1 ms] Creating account in Unix KDC
[ 29] Successfully connected to ip x.x.x.x, port 749
using TCP
**[ 45] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:createVifKrbAccountUsingKadmin line:223
**[ 45] FAILURE: Failed to create spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
** Invalid argument
[ 45] Uncaptured failure while creating account

The /var/log/kadmind.log is also fairly useless here:

Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Principal does not exist, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x
Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_create_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid argument, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

ONTAP is just reporting what kadmin says!

Our clue comes from when we try to manually create the SPN using kadmin:

kadmin: add_principal -randkey nfs/server.domain.com@DOMAIN.COM
WARNING: no policy specified for nfs/server.domain.com@DOMAIN.COM; defaulting to no policy
add_principal: Kerberos database constraints violated while creating "nfs/server.domain.com@DOMAIN.COM".

Same goes for modify_principal:

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
(vserver nfs kerberos interface enable)

Username: ontap

Password:

Warning: An account that matches the given service principal name already exists. Re-using this account deletes and re-creates the account if it is not shared by the LIFs in Vserver "NFS". This
invalidates any existing Kerberos service tickets and keys for this account. Do you want to re-use this account? {y|n}: y

Error: NFS Kerberos bind SPN procedure failed
[ 0 ms] Creating account in Unix KDC
[ 39] Successfully connected to ip x.x.x.x, port 749
using TCP
[ 45] Re-using the account for
spn=nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
**[ 53] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:randomizePasswordUsingKadmin line:287
**[ 53] FAILURE: Failed to randomize password for spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'ontap@CENTOS-LDAP.LOCAL'
[ 54] Uncaptured failure while creating account


Mar 24 11:14:54 centos8-ipa.centos-ldap.local kadmind[17208](Notice): Request: kadm5_randkey_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid key/salt tuples, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

And delete_principal:

::*> kerberos interface disable -vserver NFS -lif ipa-krb
(vserver nfs kerberos interface disable)

Username: ontap

Password:

Warning: This command deletes the service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" from the machine account on the KDC. Do you want to continue? {y|n}: y

Error: command failed: Failed to disable NFS Kerberos on LIF "ipa-krb". Failed to delete the account associated with the Kerberos service principal name. Reason: cifs smb kadmin error.

So that means, even though we gave full rights to that user in the kadm5.acl file, we can’t create principals in FreeIPA using kadmin. They want to funnel you to use the IPA tools – likely for a good reason. There’s probably a bunch of other automated steps that take place with those tools, so this is in the interest of simplicity. and security.

How do we get past this?

There are two options here.

1. Open up the permissions on FreeIPA to use kadmin to create/add/modify/delete principals. (I haven’t figured out how to do this yet)
2. Manually create the SPN and keytab file and then use ONTAP to transfer the keytab via URI

I went with option 2.

The TL;DR steps, as provided by Alexander Bokovoy in the comments:

- kinit admin
- ipa host-add demo-ipa.centos-ldap.local
- ipa service-add nfs/demo-ipa.centos-ldap.local
- ipa-getkeytab -p nfs/demo-ipa.centos-ldap.local -k ./nfs.keytab -e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

Remember that ONTAP only supports the following enctypes for NFS Kerberos:

• AES-128 and AES-256
• DES and 3DES

Then, you copy that file to your HTTP or FTP server. The address to the file will be used in the ONTAP CLI command.

Note: If you’re using IIS, you may have to allow .keytab as a MIME type.

For example:

Once the file is web-accessible, you run the kerberos interface enable command and use the -keytab-uri option to upload the keytab. Unsupported enctypes will get discarded.

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL -keytab-uri http://web-server/files/ipakrb-ontap.keytab
(vserver nfs kerberos interface enable)

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

Warning: Keys for encryption types "des-cbc-crc,des3-cbc-sha1,aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96" are required for Vserver "NFS" but found keys only for encryption types
"aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96". Keys for encryption types "des-cbc-crc,des3-cbc-sha1" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" are
missing. Available keys will be imported. Do you want to continue? {y|n}: y

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

You’ll also want to ensure you have a valid UNIX user or name mapping rule for the krb-spn name mapping for the NFS clients. That’s covered in TR-4616 in detail.

Once that’s all done, NFS Kerberos mounts should work fine!

[root@centos8-1 sysconfig]# mount -o nfsvers=4,minorversion=1,sec=krb5 x.x.x.x:/kerberos /mnt
[root@centos8-1 sysconfig]# mount | grep krb
x.x.x.x:/kerberos on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.y,local_lock=none,addr=x.x.x.x)
# su ipa-user
sh-4.4$ cd /mnt
sh: cd: /mnt: Permission denied
sh-4.4$ kinit
Password for ipa-user@CENTOS-LDAP.LOCAL:
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ cd /mnt
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
03/23/2020 14:40:16 03/24/2020 14:39:27 nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ touch krb5file
sh-4.4$ ls -la
total 4
drwxrwxrwx. 2 root root 4096 Mar 23 14:40 .
dr-xr-xr-x. 17 root root 224 Jan 21 10:03 ..
-rw-------. 1 ipa-user admins 0 Mar 23 14:40 krb5file
-rw-r--r--. 1 root root 0 Mar 23 14:15 v3nokrb
-rw-r--r--. 1 root root 0 Mar 23 14:16 v4nokrb
-rw-------. 1 ipa-user admins 0 Mar 23 14:22 v4nokrb-ipa

I opened a bug to make this easier in ONTAP with FreeIPA (1308667), so if you run into this, open a case and have it attached to the bug.

If you have questions, suggestions or get stuck (or if I got something wrong), comment below!

Using Windows Lightweight Directory Services for UNIX Identity Management with ONTAP

Windows Active Directory domains have been the way to leverage UNIX identity management in environments using Windows, given the tight integration with Kerberos, Windows accounts and ease of use. I cover a lot of this in TR-4073 (with a new LDAP-only TR coming out soon).

But, it doesn’t always fit all use cases.

For example, what if:

• You don’t have a Windows Active Directory domain?
• You don’t have access or permission to modify Active Directory domain accounts to use UNIX attributes?
• You don’t need a full-fledged AD domain with hundreds or thousands of users/groups, but only need a handful of users and groups for a single application?

There are likely other use cases that could apply here, but if you need LDAP without dealing with AD domains, we can leverage something called Windows Active Directory Lightweight Directory Services for identity management and LDAP services.

This also illustrates that pretty much *any* LDAP server can be used with ONTAP.

For example, this is a query from ONTAP to a server running AD LDS:

::*> getxxbyy getgrlist -node node1 -vserver NFS -username lds
(vserver services name-service getxxbyyy getgrlist)
pw_name: lds
Groups: 1101 1102


::*> getxxbyyy getpwbyname -node node1 -vserver NFS -username lds
(vserver services name-service getxxbyyy getpwbyname)
pw_name: lds
pw_passwd:
pw_uid: 1001
pw_gid: 1101
pw_gecos:
pw_dir: 
pw_shell:

The configuration from ONTAP is identical for any LDAP server. In my case, I used the read-only LDAP client schema template named “MS-AD-BIS.”

Basic configuration steps:

• Create the LDAP client (“ldap client create” using the stock LDAP template MS-AD-BIS)
• Create/enable LDAP on the SVM (ldap create)
• Configure DNS on the SVM (dns create/modify)
• Modify ns-switch to use LDAP (ns-switch modify)

This was my LDAP client configuration:

::*> ldap client show -client-config LDS

Vserver: NFS
Client Configuration Name: LDS
LDAP Server List: PARISI-WIN2019
(DEPRECATED)-LDAP Server List: -
Active Directory Domain: -
Preferred Active Directory Servers: -
Bind Using the Vserver's CIFS Credentials: false
Schema Template: MS-AD-BIS
LDAP Server Port: 389
Query Timeout (sec): 3
Minimum Bind Authentication Level: anonymous
Bind DN (User): administrator
Base DN: CN=LDS-LDAP,DC=PARISI-WIN2019
Base Search Scope: subtree
User DN: -
User Search Scope: subtree
Group DN: -
Group Search Scope: subtree
Netgroup DN: -
Netgroup Search Scope: subtree
Vserver Owns Configuration: false
Use start-tls Over LDAP Connections: false
Enable Netgroup-By-Host Lookup: false
Netgroup-By-Host DN: -
Netgroup-By-Host Scope: subtree
Client Session Security: none
LDAP Referral Chasing: false
Group Membership Filter:

The hardest part of this configuration for me was the AD LDS side, as there are a bunch of manual steps involved.

Configuring AD LDS for LDAP Services

The best guide I came across was this one:

AD LDS Identity Mapping for Services for NFS

Even with this guide, however, there were a few tweaks that needed to be made to the steps. For example, in the guide, they used the AD Schema Manager. But since this isn’t a full AD instance, you might be stuck using ADSI Edit.

One place I also got stuck was a RTFM moment where I forgot in my initial configuration to create an application partition. This is essential, as this is where you will be creating the OUs/Containers, users and groups for the LDAP services.

When creating the new objects, you’ll want to consider populating the following fields in the users/groups for use with ONTAP (based on the MS-AD-BIS default schema).

Already populated at creation

When you create a new object, “cn” gets autopopulated. This is used to pull the Group names. (User names use “uid” by default)

• cn

Required

“Required” here means that ONTAP won’t be able to query for the object without these attributes populated properly.

• uid
• uidNumber
• gidNumber

Optional, but recommended

These attributes are recommended, but things will mostly work without setting them (unless you require proper group memberships).

• unixHomeDirectory
• memberUid or member (for group memberships)

Optional for specific use cases

This is basically only if you need things like netgroup services or name mapping to Windows users.

• sAMAccountName (for asymmetric name mappings to Windows)
• nisObject
• nisMapName
• nisMapEntry
• nisNetgroupTriple
• memberNisNetgroup
• loginShell (would need to be added manually)

User and Group creation

For user and group management/creation, you can either use ADSI Edit or PowerShell cmdlets. The ADSI Edit method is covered in the linked guide above.

For PowerShell you can use the standard Get-ADuser/Group and New-ADUser/Group. The catch is that you may have to specify the server – especially if your LDS server is a member of the AD domain.

For example:

PS C:\> get-aduser -Identity lds -server PARISI-WIN2019 -Properties uid,uidNumber,gidNumber,gecos,unixHomeDirectory

DistinguishedName : CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
Enabled : False
gecos : AD LDS User
gidNumber : 1101
GivenName :
Name : lds
ObjectClass : user
ObjectGUID : 087692ff-ae7f-4171-922a-98accfdfdaa8
SID : S-1-396492173-265181619-2703889971-1168526272-875569285-466579950
Surname :
uid : {lds}
uidNumber : 1001
unixHomeDirectory : /u/lds
UserPrincipalName : lds@NTAP.LOCAL

PS C:\> Get-ADGroup -Identity ldsgroup1 -server PARISI-WIN2019 -Properties gidNumber,member,memberUid

DistinguishedName : CN=ldsgroup1,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
gidNumber         : 1101
GroupCategory     : Security
GroupScope        : Global
member            : {CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019}
Name              : ldsgroup1
ObjectClass       : group
ObjectGUID        : 5764424e-aba4-4e68-bff5-b7a6989d3d0c
SID               : S-1-396492173-265181619-2048905208-1111646721-233999250-1998119334

PS C:\> Get-ADGroup -Identity ldsgroup2 -server PARISI-WIN2019 -Properties gidNumber,member,memberUid

DistinguishedName : CN=ldsgroup2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
gidNumber         : 1102
GroupCategory     : Security
GroupScope        : Global
member            : {CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019}
Name              : ldsgroup2
ObjectClass       : group
ObjectGUID        : 8aec090c-0865-4b80-bd4a-723884524ff6
SID               : S-1-396492173-265181619-3623710820-1184275431-3423871139-2897951880

Note that in the group examples, I used “member” – this enables ONTAP to use RFC-2307bis. When you add members using “member,” you use the “Add DN” button in ADSI Edit or the Add-ADGroupMember PowerShell cmdlet. Then you use the full DN of the user.

For example:

PS C:\> Add-ADGroupMember -Identity ldsgroup2 -Members "CN=lds2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019" -Server PARISI-WIN2019

PS C:\> Get-ADGroup -Identity ldsgroup2 -server PARISI-WIN2019 -Properties member

DistinguishedName : CN=ldsgroup2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
GroupCategory : Security
GroupScope : Global
member : {CN=lds2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019,
CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019}
Name : ldsgroup2
ObjectClass : group
ObjectGUID : 8aec090c-0865-4b80-bd4a-723884524ff6
SID : S-1-396492173-265181619-3623710820-1184275431-3423871139-2897951880

Now ONTAP will be able to query for both groups for that user:

::*> getxxbyy getgrlist -node node1 -vserver NFS -username lds2
(vserver services name-service getxxbyyy getgrlist)
pw_name: lds2
Groups: 1101 1102

So there it is – ONTAP using AD LDS for UNIX Identity Management.

Behind the Scenes: Episode 233 – NetApp Acquires Talon Storage Solutions

Welcome to the Episode 233, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

This week on the podcast, we break down NetApp’s acquisition of Talon Storage Solutions and how it fits into NetApp’s Data Fabric Strategy. We cover why the acquisition was made, what to expect for new and old Talon customers, and what this means for NetApp’s FlexCache feature.

For more information on Talon:

https://www.talonstorage.com/

For information on FlexCache:

https://www.netapp.com/us/media/tr-4743.pdf

Guests include:

• Anthony Lye (@AnthonyLye, anthony.lye@netapp.com), Senior Vice President – NetApp Cloud Data Services
• Shirish Phatak (https://www.linkedin.com/in/phatak), Senior Director – NetApp/Founder – Talon
• Chris Hurley (@averageguyX), FlexCache TME – NetApp https://averageguyx.blogspot.com/

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 233: NetApp Acquires Talon Storage Solutions – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 232 – International Women’s Day 2020 at NetApp

Welcome to the Episode 232, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

This week on the podcast, we welcome some of NetApp’s best and brightest women to discuss how NetApp celebrated International Women’s Day 2020.

Joining us on the podcast:

• Amy Welsh (@amyrwelsh), Director of NetApp Professional Services
• Rachelle Helton (@rachelle_helton), Virtual Sales Strategic Programs and Operations Manager
• Helen Kaleekal (https://www.linkedin.com/in/helenkaleekal/), Director Finance Transformation Office
• Aruna Ghildiyal (https://www.linkedin.com/in/aruna-gairola-ghildiyal-8660543/), Global Quality and Environmental Management Systems Manager

For more information about NetApp’s Women in Technology program, follow @NetAppWIT on Twitter or visit wit.netapp.com if you’re internal to NetApp.

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 232: International Women’s Day at NetApp – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 231 – How NetApp Automates Lab on Demand

Welcome to the Episode 231, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

This week on the podcast, NetApp’s Lab on Demand director Toby Creek (@NTAPtoby) joins us to discuss how Lab on Demand has handled increasing scale, public access and automation of lab environment setup.

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 231: How NetApp Automates Lab on Demand – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here: