Recently, I was working on a project where I was creating a POC of an Ubuntu container that could authenticate to LDAP and mount NFS Kerberos mounts without any interaction that would eventually be used in a Kubernetes environment. It was an improvement on the container image I created a while back in “Securing NFS mounts in a Docker container,” and I’ll write more about it in another post later.
But, since it’s almost Christmas, I wanted to deliver a few gifts I discovered during this process that can help with other deployments.
KSU
No, I didn’t discover Kansas State University.
What I *did* discover was a way to “su” as a user while also running kinit at the same time. The ksu utility allows you to do that with the following:
# ksu username -n username
When you run that, you “su” as the UID and kinit kicks off:
root@cfeac39a405f:/# ksu student1 -n student1 WARNING: Your password may be exposed if you enter it here and are logged in remotely using an unsecure (non-encrypted) channel. Kerberos password for student1@NTAP.LOCAL: : Changing uid to student1 (1301)
And this is the ticket (NFS service ticket came with the autofs mount of the homedir):
student1@cfeac39a405f:/$ klist Ticket cache: FILE:/tmp/krb5cc_1301.kufVdH57 Default principal: student1@NTAP.LOCAL Valid starting Expires Service principal 12/23/21 12:04:15 12/23/21 13:04:15 krbtgt/NTAP.LOCAL@NTAP.LOCAL renew until 12/30/21 12:04:15 12/23/21 12:04:16 12/23/21 13:04:15 nfs/demo.ntap.local@NTAP.LOCAL renew until 12/30/21 12:04:15
student1@cfeac39a405f:/$ mount | grep home auto.home on /home type autofs (rw,relatime,fd=18,pgrp=118,timeout=50,minproto=5,maxproto=5,indirect,pipe_ino=1359512)
Pretty cool, but not as cool as the next thing I discovered…
msktutil
Getting Kerberos keytabs on Linux clients has traditionally been a pain – especially when using Active Directory as the KDC. Commands like “net ads” and “realm join” have made this simpler, but they usually require admin interaction and with containers, you can’t always do that. Nor would you really want to – who wants to join every container you create to the domain?
For Kerberos mounts, a machine account keytab is used for the initial mount to the ONTAP NFS server and the SPN you create will map into ONTAP and try to authenticate to a valid UNIX user that the ONTAP SVM has to know about. In AD KDCs, when you create a keytab, you have to map it to a user, which then populate the userPrincipalName field, which is then passed to ONTAP as the incoming SPN.
With CVS/ANF instances of ONTAP, you are at the mercy of whatever the instance has configured, so you need to either configure the SPN to MACHINE$@DOMAIN.COM or root/fqdn.domain.com@DOMAIN.COM for things to work without any other configuration.
In AD, there’s a utility called “ktpass,” which works fine, but then you have to worry about transferring the keytab file to the Linux client every time you change it and if you want to rotate keytabs on a regular basis, this gets to be cumbersome.
What I like to do with containers is have them all share a common keytab for their LDAP bind with SSSD and the NFS mount authentication. Regular users won’t be able to access the mount without using kinit anyway, and most containers don’t require a huge level of security on the container side. Kerberos is usually just there to encrypt the traffic in flight.
Since we need a keytab for containers to share located in a common location and we don’t really want to share the same keytab as the host, we can’t necessarily join the domain with “realm.” And since we’re not AD domain admins with regular access to the KDCs, ktpass is hard to manage as well.
The msktutil tool allows interaction with the KDC for keytab creation on the local Linux machine, as well as keytab updates if you want to periodically refresh the keytabs (which you should, for better overall security).
All you need to do on the Linux client is kinit as a user with permissions on the KDC to create objects. Then, when you run the “msktutil create” command, it will create a machine account in AD and a keytab based on your desired parameters. This sample command creates a keytab with AES-256 encryption and a SPN/UPN of root/fqdn:
If I don’t want extra entries, I can delete the old keytab and re-create it. Once you rotate the keytab, you may have to re-build the container to ensure the keytab file is updated in the container.
Restart services when a container starts
One challenge I faced when setting these up was that the LDAP and NFS services didn’t always work properly when you first start the container. So I found myself having to manually restart the services. When you’re running a container, the idea is that you shouldn’t have to babysit them. So, I found that creating a simple shell script to restart services, like this:
#!#!/bin/sh
sudo service dbus start
sudo /sbin/rpcbind
sudo service nfs-common restart
sudo /usr/sbin/rpc.gssd
sudo /usr/sbin/rpc.svcgssd
sudo service sssd start
sudo service autofs start
sudo service sssd restart
That script gets copied during the container run with these lines in the dockerfile:
# Script to start services
COPY configure-nfs-ubuntu.sh /usr/local/bin/configure-nfs-ubuntu.sh
RUN chmod +x /usr/local/bin/configure-nfs-ubuntu.sh
And then adding this line to the end of .bashrc did the trick:
sh /usr/local/bin/configure-nfs-ubuntu.sh
And this is how it looks when the container starts:
# docker exec -it containername bash
* system message bus already started; not starting.
rpcbind: another rpcbind is already running. Aborting
* Stopping NFS common utilities [ OK ]
* Starting NFS common utilities [ OK ]
* Starting automount...
That’s all I have for now. Stay tuned for more on this topic, including updates to the “This is the Way” series detailing my first installation of Kubernetes. Feel free to comment below!
When I’m testing stuff out for customer deployments that I don’t work with a ton, I like to keep notes on the work so I can reference it later for TRs or other things. A blog is a great place to do that, as it might help other people in similar scenarios. This won’t be an exhaustive list, and it is certain to change over time and possibly make its way into a TR, but here we go…
ONTAP and Multiprotocol NAS
Before I get into the MacOS client stuff, we need to understand ONTAP multiprotocol NAS, as it can impact how MacOS clients behave.
In ONTAP, you can serve the same datasets to clients regardless of the NAS protocol they use (SMB or NFS). Many clients can actually do both protocols – MacOS is one of those clients.
The way ONTAP does Multiprotocol NAS (and keeps permissions predictable) is via name mappings and volume “security styles,” which controls what kind of ACLs are in use. TR-4887 goes into more detail on how all that works, but at a high level:
NTFS security styles use NTFS ACLs
SMB clients will map to UNIX users and NFS clients will require mappings to valid Windows users for authentication. Then, permissions are controlled via ACLs. Chmod/chown from NFS clients will fail.
SMB clients will require mappings to a valid UNIX user for permissions; NFS clients will only require mapping to a UNIX user name if using NFSv4 ACLs. SMB clients can do *some* permissions changes, but on a very limited basis.
Mixed security styles always use either UNIX or NTFS effective security styles, based on last ACL change
Basically, if an NFS client chmods a file, it switches to UNIX security style. If an SMB client changes ownership of the file, it flips back to NTFS security style. This allows you to change permissions from any client, but you need to ensure you have proper name mappings in place to avoid undesired permission behavior. Generally, we recommend avoiding mixed security styles in most cases.
MacOS NFS Client Considerations
When using MacOS for an NFS client, there are a few things I’ve run into the past week or two while testing that you would want to know to avoid issues.
MacOS can be configured to use Active Directory LDAP for UNIX Identities
When you’re doing multiprotocol NAS (even if the clients will only do NFS, your volumes might have NTFS style permissions), you want to try to use a centralized name service like LDAP so that ONTAP, SMB clients and NFS clients all agree on who the users are, what groups they belong to, what numeric IDs they have, etc. If ONTAP thinks a user has a numeric ID of 1234 and the client things that user has a numeric ID of 5678, then you likely won’t get the access you expected. I wrote up a blog on configuring MacOS clients to use AD LDAP here:
MacOS clients can also be configured to use single sign on with AD and NFS home directories
Your MacOS clients – once added to AD in the blog post above – can now log in using AD accounts. There’s also an additional tab in the Directory Utility that allows you to auto-create home directories when a new user logs in to the MacOS client.
But you can also configure the auto-created home directories to leverage an NFS mount on the ONTAP storage system. You can configure the MacOS client to automount homedirs and then configure the MacOS client to use that path. (This process varies based on Mac version; I’m on 10.14.4 Catalina)
By default, the homedir path is /home in auto_master. We can use that.
Then, chmod the /etc/auto_home file to 644:
$ sudo chmod 644 /etc/auto_home
Create a volume on the ONTAP cluster for the homedirs and ensure it’s able to be mounted from the MacOS clients via the export policy rules (TR-4067 covers export policy rules):
Now, when I cd to /home/username, it automounts that path:
$ cd /home/prof1 $ mount demo:/machomedirs/prof1 on /home/prof1 (nfs, nodev, nosuid, automounted, nobrowse)
But if I want that path to be the new homedir path, I would need to log in as that user and then go to “System Preferences -> Users and Groups” and right click the user. Then select “Advanced Options.”
Then you’d need to restart. Once that happens, log in again and when you first open Terminal, it will use the NFS homedir path.
NOTE: You may want to test if the Mac client can manually mount the homedir before testing logins. If the client can’t automount the homedir on login things will break.
Alternately, you can create a user with the same name as the AD account and then modify the homedir path (this removes the need to login). The Mac will pick up the correct UID, but the group ID may need to be changed.
If you use SMB shares for your home directories, it’s as easy as selecting “Use UNC path” in the User Experience area of Directory Utility (there’s no way to specify NFS here):
With new logins, the profile will get created in the qtree you created for the homedir (and you’ll go through the typical initial Mac setup screens):
When you open terminal, it automounts the NFS home directory mount for that user and drops you right into your folder!
Mac NFS Considerations, Caveats, Issues
If you’re using NFS on Mac clients, there are two main things to remember:
Volumes/qtrees using UNIX security styles work best with NFS in general
Terminal/CLI works better than Finder in nearly all instances
If you have to/want to use Finder, or you have to/want to use NTFS security styles for multiprotocol, then there are some things you’d want to keep in mind.
If possible, connect the Mac client to the Active Directory domain and use LDAP for UNIX identities as described above.
Ensure your users/groups are all resolving properly on the Mac clients and ONTAP system. TR-4887 and TR-4835 cover some commands you can use to check users and groups, name mappings, group memberships, etc.
If you’re using NTFS security style volumes/qtrees and want the Finder to work properly for copies to and from the NFS mount, configure the NFS export policy rule to set -ntfs-unix-security-ops to “ignore” – Finder will bail out if ONTAP returns an error, so we want to silently fail those operations (such as SETATTR; see below).
When you open a file for reading/writing (such as a text file), Mac creates a ._filename file along with it. Depending on how many files you have in your volume, this can be an issue. For example, if you open 1 million files and Mac creates 1 million corresponding ._filename files, that starts to add up. Don’t worry! You’re not alone: https://apple.stackexchange.com/questions/14980/why-are-dot-underscore-files-created-and-how-can-i-avoid-them
One issue that I did a pretty considerable amount of analysis on was the aforementioned “can’t copy using Finder.” Here are the dirty details…
Permissions Error When Copying a File to a NFS Mount in ONTAP using Finder
In this case, a file copy worked using Terminal, but was failing with permissions errors when using Finder and complaining about the file already existing.
First, it wants a login (which shouldn’t be needed):
Then it says this:
If you select “Replace” this is the error:
If you select “Stop” it stops and you are left with an empty 0 byte “file” – so the copy failed.
If you select “Keep Both” the Finder goes into an infinite loop of 0 byte file creations. I stopped mine at around 2500 files (forced an unmount):
# ls -al | wc -l
1981
# ls -al | wc -l
2004
# ls -al | wc -l
2525
So what does that happen? Well, in a packet trace, I saw the following:
The SETATTR fails on CREATE (expected in NFS operations on NTFS security style volumes in ONTAP, but not expected for NFS clients as per RFC standards):
181 60.900209 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 182), DH: 0x8ec2d57b/copy-file-finder << Mac NFS client checks if the file exists
182 60.900558 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 181) Error: NFS3ERR_NOENT << does not exist, so let’s create it!
183 60.900633 x.x.x.x x.x.x.y NFS 238 V3 CREATE Call (Reply In 184), DH: 0x8ec2d57b/copy-file-finder Mode: EXCLUSIVE << creates the file
184 60.901179 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 183)
185 60.901224 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 186), FH: 0x7b82dffd
186 60.901564 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 185) Error: NFS3ERR_PERM << fails setting attributes, which also fails the copy of the actual file data, so we have a 0 byte file
Then it REMOVES the file (since the initial operation fails) and creates it again, and SETATTR fails again. This is where that “Keep Both” loop behavior takes place.
Since Finder bails on the error, setting the NFS server to return no error here for this export (ntfs-unix-security-ops ignore) on this client allows the copy to succeed. You can create granular rules in your export policy rules to just set that option for your Mac clients.
Now, why do our files all show as 777?
Displaying NTFS Permissions via NFS
Because NFS doesn’t understand NTFS permissions, the job to translate user identities into valid access rights falls onto the shoulders of ONTAP. A UNIX user maps to a Windows user and then that Windows user is evaluated against the folder/file ACLs.
So “777” here doesn’t mean we have wide open access; we only have access based on the Windows ACL. Instead, it just means “the Linux client can’t view the access level for that user.” In most cases, this isn’t a huge problem. But sometimes, you need files/folders not to show 777 (like for applications that don’t allow 777).
In that case, you can control somewhat how NFS clients display NTFS ACLs in “ls” commands with the NFS server option ntacl-display-permissive-perms.
[-ntacl-display-permissive-perms {enabled|disabled}] - Display maximum NT ACL Permissions to NFS Client (privilege: advanced)
This optional parameter controls the permissions that are displayed to NFSv3 and NFSv4 clients on a file or directory that has an NT ACL set. When true, the displayed permissions are based on the maximum access granted by the NT ACL to any user. When false, the displayed permissions are based on the minimum access granted by the NT ACL to any user. The default setting is false.
The default setting of “false” is actually “disabled.” When that option is enabled, this is the file/folder view:
When that option is disabled (the default):
This option is covered in more detail in TR-4067, but it doesn’t require a remount to take effect. It may take some time for the access caches to clear to see the results, however.
Keep in mind that these listings are approximations of the access as seen by the current user. If the option is disabled, you see the minimum access; if the option is enabled, you see the maximum access. For example, the “test” folder above shows 555 when the option is disabled, but 777 when the option is enabled.
These are the actual permissions on that folder:
::*> vserver security file-directory show -vserver DEMO -path /FG2/test
Vserver: DEMO
File Path: /FG2/test
File Inode Number: 10755
Security Style: ntfs
Effective Style: ntfs
DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
UNIX User Id: 1102
UNIX Group Id: 10002
UNIX Mode Bits: 777
UNIX Mode Bits in Text: rwxrwxrwx
ACLs: NTFS Security Descriptor
Control:0x8504
Owner:BUILTIN\Administrators
Group:NTAP\ProfGroup
DACL - ACEs
ALLOW-Everyone-0x1200a9-OI|CI (Inherited)
ALLOW-NTAP\prof1-0x1f01ff-OI|CI (Inherited)
So, prof1 has Full Control (7) and “Everyone” has Read (5). That’s where the minimum/maximum permissions show up. So you won’t get *exact* permissions here. If you want exact permission views, consider using UNIX security styles.
DS_Store files
Mac will leave these little files laying around as users browse shares. In a large environment, that can start to create clutter, so you may want to consider disabling the creation of these on network shares (such as NFS mounts), as per this:
In NetApp ONTAP, it’s possible to serve data to NAS clients over SMB and NFS, including the same datasets. This is known as “multiprotocol NAS” and I cover the best practices for that in the new TR-4887:
When you do multiprotocol NAS in ONTAP (or really, and storage system), it’s usually best to leverage a centralized repository for user names, group names and numeric IDs that the NAS clients and NAS servers all point to (such as LDAP). That way, you get no surprises when accessing files and folders – user and groups get the expected ownership and permissions.
One of the more commonly implemented solutions for LDAP in environments that serve NFS and SMB is Active Directory. In these environments, you can either use either native UNIX attributes or a 3rd party utility, such as Centrify. (Centrify basically just uses the AD UNIX attributes and centralizes the management into a GUI – both are covered in the LDAP TR.)
While most Linux clients are fairly straightforward for LDAP integration, MacOS does things slightly differently. However, it’s pretty easy to configure.
Note: The steps may vary based on your environment configs and this covers just AD LDAP; not OpenLDAP/IDM or other Linux-based LDAP.
Step 1: Ensure the Mac is configured to use the same DNS as the AD domain
This is done via “Network” settings in System Preferences. DNS is important here because it will be used to query the AD domain when we bind the Mac for the Directory Services. In the following, I’ve set the DNS server IP and the search domain to my AD domain “NTAP.LOCAL”:
Then I tested the domain lookup in Terminal:
Step 2: Configure Directory Services to use Active Directory
The “Directory Utility” is what we’ll use here. It’s easiest to use the spotlight search to find it.
Essentially, this process adds the MacOS to the Active Directory domain (as you would with a Windows server or a Linux box with “realm join”) and then configures the LDAP client on the Mac to leverage specific attributes for LDAP queries.
In the above, I’ve used uidNumber and gidNumber as the attributes. You can also control/view these via the CLI command “dsconfigad”:
I can see in my Windows AD domain the machine account was created:
A few caveats here about the default behavior for this:
LDAP queries will be encrypted by default, so if you’re trying to troubleshoot via packet capture, you won’t see a ton of useful info (such as attributes used for queries). To disable this (mainly for troubleshooting purposes):
MacOS uses sAMAccountName as the user name/uid value, so it should work fine with AD out of the gate
MacOS adds additional Mac-specific system groups to the “id” output (such as netaccounts:62, and GIDs 701/702); these may need to be added to LDAP, depending on file ownership
LDAP queries to AD from Mac will use the Global Catalog port 3268 by default when using Active Directory config (which I was able to see from a packet capture)
This use of the global catalog port can be problematic, as standard LDAP configurations in AD don’t set the Global Catalog to replicate attributes, but rather uses the standard port 389/636 for LDAP communication. AD doesn’t replicate the UNIX attributes across the global catalog by default for LDAP, so you’d have to configure that manually (covered in TR-4835) or modify the port the Mac uses for LDAP.
My AD domain does have the attributes that replicate via the Global Catalog, so the LDAP lookups work for me from the Mac:
Here’s what the prof1 user looks like from a CentOS client:
# id prof1 uid=1102(prof1) gid=10002(ProfGroup) groups=10002(ProfGroup),10000(Domain Users),48(apache-group),1101(group1),1202(group2),1203(group3)
Most people aren’t going to want/be allowed to crack open ADSIEdit and modify schema attributes, so you’d want to change how MacOS queries LDAP to use port 389 or 636. I’m currently waiting on word of how to do that from Apple, so I’ll update when I get that info. If you are reading this and already know, feel free to add to the comments!
Step 3: Mount the NetApp NFS export and test multiprotocol access
NFS mounts to UNIX security style volumes are pretty straightforward, so we won’t cover that here. Where it gets tricky is when your ONTAP volumes are NTFS security style. When that’s the case, a UNIX -> Windows name mapping occurs when using NFS, as we need to make sure the user trying to access the NTFS permissions truly has access.
This is the basic process:
MacOS NFS client sends a numeric UID and GID to the NetApp ONTAP system (if NFSv3 is used)
If the volume is NTFS security, ONTAP will try to translate the numeric IDs into user names. The method depends on the cluster config; in this case, we’ll use LDAP.
If the numeric IDs map to user names/group names, then ONTAP uses those UNIX names and tries to find a valid Windows name with the same names; if none exist, ONTAP looks for explicit name mapping rules and a default Windows user; if none of those work then access is denied.
I have mounted a volume to my MacOS client that uses NTFS security style.
Error: Get user credentials procedure failed [ 33] Determined UNIX id 501 is UNIX user 'Podcast' [ 34] Using a cached connection to ntap.local [ 36] Trying to map 'Podcast' to Windows user 'Podcast' using implicit mapping [ 36] Successfully connected to ip 10.193.67.236, port 445 using TCP [ 46] Successfully authenticated with DC oneway.ntap.local [ 49] Could not find Windows name 'Podcast' [ 49] Unable to map 'Podcast'. No default Windows user defined. **[ 49] FAILURE: Name mapping for UNIX user 'Podcast' failed. No ** mapping found
Error: command failed: Failed to get user credentials. Reason: "SecD Error: Name mapping does not exist".
So when I try to access this mount, it will attempt to use UID 501 and translate it to a UNIX user and then to a Windows user. Since ONTAP can’t translate UID 501 to a valid Windows user, this will fail and we’ll see it in the event log of the ONTAP CLI.
Here’s the access failure:
Here’s the ONTAP error:
ERROR secd.nfsAuth.noNameMap: vserver (DEMO) Cannot map UNIX name to CIFS name. Error: Get user credentials procedure failed [ 33] Determined UNIX id 501 is UNIX user 'Podcast' [ 34] Using a cached connection to ntap.local [ 36] Trying to map 'Podcast' to Windows user 'Podcast' using implicit mapping [ 36] Successfully connected to ip 10.193.67.236, port 445 using TCP [ 46] Successfully authenticated with DC oneway.ntap.local [ 49] Could not find Windows name 'Podcast' [ 49] Unable to map 'Podcast'. No default Windows user defined. **[ 49] FAILURE: Name mapping for UNIX user 'Podcast' failed. No mapping found
When I “su” to a user that *does* have a valid Windows user (such as prof1), this works fine and I can touch a file and get the proper owner/group:
Note that in the above, we see “root:wheel” owned folders; just because root is disabled by default on MacOS doesn’t mean that MacOS isn’t aware of the user. Those folders were created on a separate NFS client.
Also, note in the above that the file shows 777 permissions; this is because those are the allowed permissions for the prof1 user on that file. The permissions are defined by Windows/NTFS. Here, they are set to “Everyone:Full Control” by way of file inheritance. These are the new permissions. Profgroup (with prof1 and studen1 as members) gets write access. Administrator gets “Full Control.” Group10 (with only student1 as a member) gets read access.
In ONTAP, you can also control the way NTFS security style files are viewed on NFS clients with the NFS server option -ntacl-display-permissive-perms. TR-4887 covers that option in more detail.
Prof1 access view after permissions change (write access):
Student1 access view after permissions change (read access only via group ACL):
Read works, but write does not (by design!)
Student2 access view (write access defined by membership in ProfGroup):
Newuser1 access view (not a member of any groups in the ACL):
Newuser1 can create a new file, however, and it shows the proper owner. The permissions are 777 because of the inherited NTFS ACLs from the share:
As you can see, we will get the expected access for users and groups on Mac NFS using NTFS security styles, but the expected *views* won’t always line up. This is because there isn’t a direct correlation between NTFS and UNIX permissions, so we deliver an approximation. ONTAP doesn’t store ACLs for both NTFS and UNIX on disk; it only chooses one or the other. If you require exact NFS permission translation via Mac NFS, consider using UNIX security style and mode bits.
Addendum: Squashing root
In the event your MacOS users enable the root account and become the “root” user on the client, you can squash the root user to an anonymous user by using ONTAP’s export policies and rules. TR-4067 covers how to do this:
And people have asked about it a few times since then. To be fair, I did say “will be a ways out…”
In actuality, I started that TR in March of 2017. And then again in February of 2019. And then started all over when the pandemic hit, because what else did I have going on? 🙂
And it’s not like I haven’t done *stuff* in that time.
The trouble was, I do multiprotocol NAS every day, so I think I had writer’s block because I didn’t know where to start and the challenge of writing an entire TR on the subject without making it 100-200 pages like some of the others I’ve written was… daunting. But, it’s finally done. And the actual content is under 100 pages!
With the COVID-19 quarantine, stay at home orders and new 1-year ONTAP release cadence, I’m finding I have a lot more spare time, which translates into time to update old, crusty technical reports!
Some of the old TRs hadn’t been updated for 3 years or so. Much of the information in those still applied, but overall, the TR either had to be retired or needed an update – if only to refresh the publish date and apply new templates.
This TR was a monolith that I wrote when I first started as a TME back in 2015-ish. It covers LDAP, Kerberos and NFSv4.x for a unified security approach to NFS. The goal was to combine everything into a centralized document, but what ended up happening was I now had a TR that was 250+ pages long. Not only is that hard to read, but it’s also daunting enough to cause people not to want to read it at all. As a result, I made it a goal to break the TR up into more manageable chunks. Eventually, this TR will be deprecated in favor of newer TRs that are shorter and more specific.
I created the NFS Kerberos TR in 2017 to focus only on Kerberos with NFS. To streamline the document, I narrowed the focus to only a set of configuration options (AD KDCs, RHEL clients, newest ONTAP version), removed extraneous details and moved examples/configuration steps to the end of the document. The end result – a 42 page document with the most important information taking up around 30 pages.
However, there hasn’t been an updated version since then. I’m currently in the process of updating that TR and was waiting on some other TRs to be completed before I finished this one. The new revision will include updated information and the page count will rise to around 60-70 pages.
This TR is another of the original documents I created and hasn’t been updated since 2017. It’s currently getting a major overhaul right now, including re-organizing the order to include the more crucial information at the start of the document and reducing the total page count by roughly 20 pages. Examples and advanced topics were moved to the back of the document and the “meat” of the TR is going to be around 90 pages.
Major changes include:
New TR template
Performance testing for NFSv3 vs. NFSv4.x
New best practice recommendations
Security best practices
Multiprotocol NAS information
Removal of Infinite Volume section
NFS credential information
As part of the TR-4073 de-consolidation project, TR-4067 will cover the NFSv4.x aspects.
This TR is nearly done and is undergoing some peer review, so stay tuned!
This TR was created to cover the DNS load balancing approaches for NAS workloads with ONTAP. It’s pretty short – 35 pages or so – and covers on-box and off-box DNS load balancing.
It was updated in May 2020 and was basically a minor refresh.
The final part of the TR-4073 de-consolidation effort was creating an independent LDAP TR. Unlike the NFS Kerberos TR, I wanted this one to cover a wide array of configurations and use cases, so the total length ended up being 135 pages, but the “meat” of the document (the most pertinent information) only takes up around 87 pages.
Windows Active Directory domains have been the way to leverage UNIX identity management in environments using Windows, given the tight integration with Kerberos, Windows accounts and ease of use. I cover a lot of this in TR-4073 (with a new LDAP-only TR coming out soon).
But, it doesn’t always fit all use cases.
For example, what if:
You don’t have a Windows Active Directory domain?
You don’t have access or permission to modify Active Directory domain accounts to use UNIX attributes?
You don’t need a full-fledged AD domain with hundreds or thousands of users/groups, but only need a handful of users and groups for a single application?
There are likely other use cases that could apply here, but if you need LDAP without dealing with AD domains, we can leverage something called Windows Active Directory Lightweight Directory Services for identity management and LDAP services.
This also illustrates that pretty much *any* LDAP server can be used with ONTAP.
For example, this is a query from ONTAP to a server running AD LDS:
Even with this guide, however, there were a few tweaks that needed to be made to the steps. For example, in the guide, they used the AD Schema Manager. But since this isn’t a full AD instance, you might be stuck using ADSI Edit.
One place I also got stuck was a RTFM moment where I forgot in my initial configuration to create an application partition. This is essential, as this is where you will be creating the OUs/Containers, users and groups for the LDAP services.
When creating the new objects, you’ll want to consider populating the following fields in the users/groups for use with ONTAP (based on the MS-AD-BIS default schema).
Already populated at creation
When you create a new object, “cn” gets autopopulated. This is used to pull the Group names. (User names use “uid” by default)
cn
Required
“Required” here means that ONTAP won’t be able to query for the object without these attributes populated properly.
uid
uidNumber
gidNumber
Optional, but recommended
These attributes are recommended, but things will mostly work without setting them (unless you require proper group memberships).
unixHomeDirectory
memberUid or member (for group memberships)
Optional for specific use cases
This is basically only if you need things like netgroup services or name mapping to Windows users.
sAMAccountName (for asymmetric name mappings to Windows)
For PowerShell you can use the standard Get-ADuser/Group and New-ADUser/Group. The catch is that you may have to specify the server – especially if your LDS server is a member of the AD domain.
Note that in the group examples, I used “member” – this enables ONTAP to use RFC-2307bis. When you add members using “member,” you use the “Add DN” button in ADSI Edit or the Add-ADGroupMember PowerShell cmdlet. Then you use the full DN of the user.
For example:
PS C:\> Add-ADGroupMember -Identity ldsgroup2 -Members "CN=lds2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019" -Server PARISI-WIN2019
PS C:\> Get-ADGroup -Identity ldsgroup2 -server PARISI-WIN2019 -Properties member
DistinguishedName : CN=ldsgroup2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
GroupCategory : Security
GroupScope : Global
member : {CN=lds2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019,CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019}
Name : ldsgroup2
ObjectClass : group
ObjectGUID : 8aec090c-0865-4b80-bd4a-723884524ff6
SID : S-1-396492173-265181619-3623710820-1184275431-3423871139-2897951880
Now ONTAP will be able to query for both groups for that user:
One question I’ve seen asked a lot, but haven’t been able to find much information on, is how to configure a Linux LDAP server (in this case, FreeIPA on CentOS/RHEL) to interact with NetApp’s Data ONTAP software to allow users to login via SSH or the web browser. With Windows Active Directory, it’s fairly simple – you set up domain tunnels and that’s about it.
So, I finally decided to try and figure it out. If you want the quick and dirty summary, scroll to the bottom.
I set up a FreeIPA server (with DNS) on CentOS 8 using the following link:
Getting ONTAP to query LDAP for users and groups after it was set up was fairly straightforward.
First, I needed DNS configured to be able to discover the LDAP server.
cluster::*> dns show -vserver cluster
Vserver: cluster
Domains: centos-ldap.local
Name Servers: 10.x.x.x
Timeout (secs): 2
Maximum Attempts: 1
Is TLD Query Enabled?: true
Require Source and Reply IPs to Match: true
Require Packet Queries to Match: true
Here’s the LDAP client configuration I set up. Keep in mind that this one wasn’t the correct config:
cluster::*> ldap client show -client-config IDM
Client Configuration Name: IDM
LDAP Server List: 10.x.x.x
(DEPRECATED)-LDAP Server List: -
Active Directory Domain: -
Preferred Active Directory Servers: -
Bind Using the Vserver's CIFS Credentials: false
Schema Template: IPA
LDAP Server Port: 389
Query Timeout (sec): 3
Minimum Bind Authentication Level: anonymous
Bind DN (User):
Base DN: dc=centos-ldap,dc=local
Base Search Scope: subtree
User DN:
User Search Scope: subtree
Group DN:
Group Search Scope: subtree
Netgroup DN: -
Netgroup Search Scope: subtree
Vserver Owns Configuration: false
Use start-tls Over LDAP Connections: false
Enable Netgroup-By-Host Lookup: false
Netgroup-By-Host DN: -
Netgroup-By-Host Scope: subtree
Client Session Security: none
LDAP Referral Chasing: false
Group Membership Filter:
I wanted to make some changes to the schema, so I went ahead and created a custom schema based on the attributes of users and groups in the LDAP server.
cluster::*> ldap client schema show -schema IPA
Schema Template: IPA
Comment:
RFC 2307 posixAccount Object Class: person
RFC 2307 posixGroup Object Class: posixgroup
RFC 2307 nisNetgroup Object Class: ipanisnetgroup
RFC 2307 uid Attribute: uid
RFC 2307 uidNumber Attribute: uidNumber
RFC 2307 gidNumber Attribute: gidNumber
RFC 2307 cn (for Groups) Attribute: cn
RFC 2307 cn (for Netgroups) Attribute: cn
RFC 2307 userPassword Attribute: userPassword
RFC 2307 gecos Attribute: gecos
RFC 2307 homeDirectory Attribute: homeDirectory
RFC 2307 loginShell Attribute: loginShell
RFC 2307 memberUid Attribute: member
RFC 2307 memberNisNetgroup Attribute: memberNisNetgroup
RFC 2307 nisNetgroupTriple Attribute: nisNetgroupTriple
Enable Support for Draft RFC 2307bis: true
RFC 2307bis groupOfUniqueNames Object Class: posixgroup
RFC 2307bis uniqueMember Attribute: member
Data ONTAP Name Mapping windowsToUnix Object Class: posixAccount
Data ONTAP Name Mapping windowsAccount Attribute: windowsAccount
Data ONTAP Name Mapping windowsToUnix Attribute: windowsAccount
No Domain Prefix for windowsToUnix Name Mapping: false
Vserver Owns Schema: false
Maximum groups supported when RFC 2307bis enabled: 256
RFC 2307 nisObject Object Class: ipahost
RFC 2307 nisMapName Attribute: nisMapName
RFC 2307 nisMapEntry Attribute: nisMapEntry
Then I ran LDAP create on the cluster SVM. If there is a configuration issue, this step will fail to prevent issues later:
cluster::> ldap check -vserver cluster
Vserver: cluster
Client Configuration Name: IDM
LDAP Status: up
LDAP Status Details: Successfully connected to LDAP server "10.x.x.x".
LDAP DN Status Details: All the configured DNs are available.
And I modified the ns-switch to use files,ldap for passwd and group:
cluster::*> ns-switch show -vserver cluster -database passwd,group
(vserver services name-service ns-switch show)
Source
Vserver Database Order
--------------- ------------ ---------
cluster group files,ldap
cluster passwd files,ldap
After setting that up, I could query users using the getXXbyYY commands in advanced privilege:
Once I got LDAP user lookups working, I went ahead and created security logins for ssh, http and ontapi services for the idm-ldap user using nsswitch as the authentication method.
cluster::*> security login show -vserver cluster -user-or-group-name idm-ldap
Vserver: cluster
Second
User/Group Authentication Acct Authentication
Name Application Method Role Name Locked Method
-------------- ----------- ------------- ---------------- ------ --------------
idm-ldap http nsswitch admin - none
idm-ldap ontapi nsswitch admin - none
idm-ldap ssh nsswitch admin - none
However, logins failed. I could see the (not very descriptive) error in the event log:
sshd.auth.loginDenied: message="Failed keyboard-interactive / pam for idm-ldap from 10.x.x.x port 58872 ssh2 "
I poked around logs for awhile but didn’t have any luck. Then, I pinged my NAS counterpart, Chris Hurley (@AverageGuyX) to see if he’d ever gotten this working before.
Luckily he had.
Turns out, there were three things going one. One I suspected to be an issue. The other two, I did not.
FreeIPA Schema – Compat vs. LDAP
So, unbeknownst to me, FreeIPA sets up two different LDAP DN trees for users. One is the normal LDAP portion. The other is for compatibility with NIS. I set the base DN to the top level of the tree, which was pulling from compat. The issue there? Compat doesn’t transfer the password hash, so there’s no way to properly authenticate. Here’s the output of the idm-ldap user. Note there is no userPassword entry in the compat entry:
To fix that, I just needed to set the user and group DN to filter where the LDAP users and groups are. Both users and groups live in cn=accounts,dc=centos-ldap,dc=local.
That’s because we’re not binding with a user that has permissions to see that field.
Directory Manager
In Linux LDAP servers, there’s a user called “Directory Manager” that has access to view everything in the schema – including userPassword fields. In my client config in ONTAP, I’m only binding as anonymous, which works fine for seeing basic user info. But for authentication, we need the password.
So, I set my bind-dn to “CN=Directory Manager,” min-bind-level to simple, and set the bind password.
If everything is set up properly, then the command works. If it’s not (such as the wrong bind DN) the command fails:
cluster::*> ldap client modify -client-config IDM -bind-dn "CN=Directory" -min-bind-level simple
Error: Validate the Ldap configuration procedure failed
[ 0 ms] Hostname found in Name Service Cache
[ 1] IP Address found in Name Service Cache
[ 1] Resolved LDAP servers: 10.x.x.x. Vserver: 9
[ 1] Failed to initiate Kerberos authentication. Trying NTLM.
[ 2] Successfully connected to ip 10.x.x.x, port 389
using TCP
[ 106] Unable to connect to LDAP (NIS & Name Mapping) service on
centos8-ipa.centos-ldap.local (Error: Invalid
credentials)
[ 106] No servers available for LDAP_NIS_AND_NAME_MAPPING,
vserver: 9, domain: .
**[ 106] FAILURE: Unable to make a connection (LDAP (NIS & Name
** Mapping):), result: 6940
Error: command failed: The LDAP client configuration "IDM" for Vservers "NFS, cluster" is an invalid configuration.
This is because ONTAP runs a config validation to ensure we don’t get silent failures due to configuration errors.
By default, FreeIPA uses a passwordStorageScheme of SSHA512, which is the most secure it offers. This is also the longest password it offers. You can view passwordStorageScheme with the following command:
Essentially, each hash provides for different password hash lengths. The longer the hash length, the more secure the password is. Naturally, CLEAR means “clear text”. When we query a user with CLEAR hashing, we see the password:
However, ONTAP can currently only use the following hashes for user authentication:
CRYPT (all flavors)
SHA
SSHA
Other flavors (like CLEAR, MD5 SHA256, SHA512, SSHA512, etc) won’t allow authentication for logins. But using CRYPT-SHA512 works. And it’s a fairly secure password hash length.
To change the FreeIPA passwordStorageScheme, we use ldapmodify:
But, most importantly, we can now login using the idm-ldap user:
# ssh idm-ldap@10.x.x.x
Password:
Last login time: 2/5/2020 23:48:55
cluster::> version
NetApp Release 9.7X19: Thu Dec 05 11:10:30 UTC 2019
As a bonus, we can also login to System Manager with that user:
Configuring LDAP Groups for Login
When I posted this blog, Twitter user @ByyvO replied about his inability to get groups working. I took that as a challenge, so here’s how to get LDAP groups working.
First of all, LDAP group memberships must be able to be queried by ONTAP properly – meaning, all groups for the user should be able to be looked up. By default, ONTAP LDAP schemas use memberUid for group membership information. FreeIPA, however, uses “member” as the attribute. It also appears to use RFC-2307bis, which is why I build a custom schema to change the group membership attribute, as well as enabling BIS. (see the schema example in the summary section below)
On the LDAP server, I created a group named “ontap-ldap” and added idm-ldap (which was used to login originally) and ipa-user.
Then, I created a security login entry for the ontap-ldap group on the cluster. In these commands I set -is-nsswitch-group to yes. If it’s set to “no” then ONTAP will treat the group name like a user – which won’t work.
I only created it for ssh and ontapi, as LDAP group auth isn’t supported yet for http:
As you can see, there is no entry here for ipa-user:
cluster::*> security login show -vserver cluster -user-or-group-name ipa-user
There are no entries matching your query.
Then, I try to login using the ipa-user. Works fine!
[root@centos8-ipa /]# ssh ipa-user@10.193.67.10
Password:
This is your first recorded login.
ontap9-tme-8040::> version
NetApp Release 9.7X19: Thu Dec 05 11:10:30 UTC 2019
Summary
Just to consolidate the lessons above into an easier to read section, here are the steps to configure LDAP authentication with FreeIPA (or other Linux LDAP servers) to manage ONTAP.
Configure DNS on the cluster SVM
Copy an LDAP schema from the RFC-2307bis template to a new schema template and modify attributes as needed
Create an LDAP client that uses “CN=Directory Manager” for the bind-dn, simple authentication as the minimum bind level, filtering user and group DN to use LDAP users and not something like “compat”
Set the bind-dn password
Create/enable LDAP on the SVM
Run LDAP check
Modify ns-switch for passwd,group databases for the cluster SVM to use files,ldap
Ensure FreeIPA is using CRYPT, SHA or SSHA hashes for passwords in ONTAP versions prior to 9.7P9; bug 1264152 is fixed in later versions
Run getXXbyYY commands to test user lookups
Create user logins with nsswitch authentication methods for the desired users
Test logins
This was the final LDAP client and schema configuration in my environment.
Believe it or not, Windows NFS is a thing. Microsoft has its own NFS server and client, which can leverage RFC compliant NFSv3 calls to a Windows Server running NFS server or to a 3rd party NFS server, such as NetApp ONTAP. It’s actually so popular, that NetApp had to re-introduce it in clustered ONTAP (it wasn’t there until ONTAP 8.2.3/8.3.1).
While Windows NFS currently provides NFSv3 clients, they don’t have NFSv4.1 clients – yet. They do provide NFSv4.1 as a server option, though:
I cover Windows NFS support in TR-4067 starting on page 116. I am bringing this topic up because it has come up again recently and I wanted to create a quick and easy blog to follow, as well as call out how you can integrate AD LDAP to help identity management.
There are a few things you have to do to get it working in ONTAP.
Specifically:
enable -v3-ms-dos-client option on the NFS server
enable -showmount on the NFS server – this prevents some weirdness with writing files
disable -enable-ejukebox and -v3-connection-drop
The command would look like this:
cluster::> set advanced
cluster::*> nfs server modify -vserver DEMO -v3-ms-dos-client enabled -v3-connection-drop disabled -enable-ejukebox false -showmount enabled
cluster::*> nfs server show -vserver DEMO -fields v3-ms-dos-client,v3-connection-drop,showmount,enable-ejukebox
vserver enable-ejukebox v3-connection-drop showmount v3-ms-dos-client
------- --------------- ------------------ --------- ----------------
DEMO false disabled enabled enabled
Once that’s done, you can mount via NFS inside Windows clients using the standard “mount” command, provided you’ve enabled the Services for UNIX functionality. There’s plenty of documentation out there for that.
Just by doing the above, here’s an example of a working NFS mount in Windows:
C:\Users\Administrator>mount DEMO:/flexvol X:
X: is now successfully connected to DEMO:/flexvol
The command completed successfully.
Here’s the cluster’s view of that connection:
ontap9-tme-8040::*> network connections active show -node ontap9-tme-8040-0* -service nfs*,mount -remote-ip 10.193.67.236
Vserver Interface Remote
CID Ctx Name Name:Local Port Host:Port Protocol/Service
--------- --- --------- ----------------- -------------------- ----------------
Node: ontap9-tme-8040-02
2968991376 4 DEMO data:2049 oneway.ntap.local:931
TCP/nfs
When I write a file to the mount, there is something that can prove to be an issue, however. Users other than Administrator will write as UID/GID of 4294967294 (-2).
ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /flexvol/student1-nfs.txt
Vserver: DEMO
File Path: /flexvol/student1-nfs.txt
File Inode Number: 1606599
Security Style: unix
Effective Style: unix
DOS Attributes: 20
DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
UNIX User Id: 4294967294
UNIX Group Id: 4294967294
UNIX Mode Bits: 755
UNIX Mode Bits in Text: rwxr-xr-x
ACLs: -
That means users won’t show up properly/as desired in UNIX NFS mounts. For example, this is that same file from CentOS:
[root@centos7 /]# cd flexvol
[root@centos7 flexvol]# ls -la | grep student1-nfs
-rwxr-xr-x 1 4294967294 4294967294 0 Feb 5 09:18 student1-nfs.txt
So, how does one fix that?
Configuring Windows NFS clients to negotiate users properly
There are a few ways to have users leverage UID/GID other than -2.
One way is to “squash” every NFS user to the same UID/GID via the old Windows standby – the Windows registry. This is useful if only a single user will be using an NFS client.
Some of the third party NFS clients (such as Cygwin and Hummingbird/OpenText) will provide local passwd and group file functionality to allow you to leverage more users. In some cases, all this does is add more registry entries.
Another was is to chmod/chown the file after it’s written. But that’s not ideal.
The best way is to leverage an existing name service (such as NIS or LDAP) and have Windows clients query for the UID and GID. If you have one already, great! It’s super easy to set up the client. Just run the following command as an administrator in cmd. My NTAP.LOCAL domain already has an LDAP server set up:
C:\Users\administrator>nfsadmin mapping WIN7-CLIENT config adlookup=yes addomain=NTAP.LOCAL
The settings were successfully updated.
Once I did that, I wrote a new file and the UID/GID was properly represented:
ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /flexvol/prof1-nfs.txt
Vserver: DEMO
File Path: /flexvol/prof1-nfs.txt
File Inode Number: 1606600
Security Style: unix
Effective Style: unix
DOS Attributes: 20
DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
UNIX User Id: 1100 UNIX Group Id: 1101
UNIX Mode Bits: 755
UNIX Mode Bits in Text: rwxr-xr-x
ACLs: -
ontap9-tme-8040::*> getxxbyyy getpwbyname -node ontap9-tme-8040-01 -vserver DEMO -username prof1
(vserver services name-service getxxbyyy getpwbyname)
pw_name: prof1
pw_passwd:
pw_uid: 1100pw_gid: 1101
pw_gecos:
pw_dir:
pw_shell:
If you’re interested, a packet trace shows that the Windows client will communicate via encrypted LDAP to query the user’s UNIX attribute information:
An added bonus of having Windows clients query LDAP for UNIX user names and groups for NFS on ONTAP is that if you’re using NTFS security style volumes, you won’t have issues connecting to those mounts.
What breaks when doing NTFS security style?
When a UNIX user attempts to access a volume with NTFS security style ACLs, ONTAP will attempt to map that user to a valid Windows user to make sure Windows ACLs can be calculated. (I cover this in Mixed perceptions with NetApp multiprotocol NAS access)
If a user comes in with the default Windows NFS ID of 4294967294 (which doesn’t translate to a UNIX user), this is what happens.
The UNIX user 4294967294 tries to access the mount.
ONTAP receives a UID of 4294967294 and attempts to map that to a Windows user
That Windows user does not exist, so access is denied. This can manifest as an error (such as when writing a file) or it could just show no files/folder.
That particular folder doeshave data. It’s just that the user can’t see it:
In ONTAP, we’d see this error, confirming that the user doesn’t exist:
2/5/2019 14:31:26 ontap9-tme-8040-02
ERROR secd.nfsAuth.problem: vserver (DEMO) General NFS authorization problem. Error: Get user credentials procedure failed
[ 15 ms] Hostname found in Name Service Cache
[ 19] Hostname found in Name Service Cache
[ 23] Successfully connected to ip 10.193.67.236, port 389 using TCP
**[ 28] FAILURE: User ID '4294967294' not found in UNIX authorization source LDAP.
[ 28] Entry for user-id: 4294967294 not found in the current source: LDAP. Ignoring and trying next available source
[ 29] Entry for user-id: 4294967294 not found in the current source: FILES. Entry for user-id: 4294967294 not found in any of the available sources
[ 44] Unable to get the name for UNIX user with UID 4294967294
With LDAP involved, access to the access to the NFS mounted volume with NTFS security works much better, because ONTAP and the client agree that user 1100 is prof1.
So, uh… what if I don’t have LDAP or NIS?
Well, in a Windows domain, you ALWAYS have an LDAP server. Active Directory leverages LDAP schemas to store information and any version of Windows Active Directory can be used to look up UNIX users and groups. In fact, the newer versions of Windows make this very easy. In older Windows versions, you had to manually extend the LDAP schema to provide UNIX attributes. Now, UNIX attributes like UID, UIDnumber, etc. are all in LDAP by default. All you have to do is populate these values with information. You can even do it via PowerShell CMDlets!
Once you have a working Active Directory LDAP environment, you can then configure ONTAP to communicate with LDAP for UNIX identities and you’re well on your way to having a scalable, functional multiprotocol NAS environment.
The one downside I’ve found with Windows NFS is that it doesn’t always play nicely when you want to use SMB on the same client. Windows gets a bit… confused. I haven’t dug into that a ton, but I’ve seen it enough to express caution. 🙂
Welcome to the Episode 137, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, we talk Name Services in ONTAP and the introduction of the new global name services cache in ONTAP 9.3 with NAS TME, Chris Hurley (@averageguyX)!
We’ll be taking next week off as we record and prepare for some big announcements coming soon!
Finding the Podcast
The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.
This week’s episode is here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
In an ONTAP cluster made up of individual nodes with individual hardware resources, it’s useful if a storage administrator can manage the entire cluster as a monolithic entity, without having to worry about what lives where.
Prior to ONTAP 9.3, name service caches were node-centric, for the most part. This sometimes could create scenarios where a cache could become stale on one node, where it was recently populated on another node. Thus, a client may get different results depending on which physical node the network connection occurred.
The following is pulled right out of the new name services best practices technical report (https://www.netapp.com/us/media/tr-4668.pdf), which acts as an update to TR-4379. I wrote some of this, but most of what’s written here is by the new NFS/Name Services TME Chris Hurley. (@averageguyx) This is basically a copy/paste, but I thought this was a cool enough feature to highlight on its own.
Global Name Services Cache in ONTAP 9.3
ONTAP 9.3 offers a new caching mechanism that moves name service caches out of memory and into a persistent cache that is replicated asynchronously between all nodes in the cluster. This provides more reliability and resilience in the event of failovers, as well as offering higher limits for name service entries due to being cached on disk rather than in node memory.
The name service cache is enabled by default. If legacy cache commands are attempted in ONTAP 9.3 with name service caching enabled, an error will occur, such as the following:
Error: show failed: As name service caching is enabled, "Netgroups" caches no longer exist. Use the command "vserver services name-service cache netgroups members show" (advanced privilege level) to view the corresponding name service cache entries.
The name service caches are controlled in a centralized location, below the name-service cache command set. This provides easier cache management, from configuring caches to clearing stale entries.
The global name service cache can be disabled for individual caches using vserver services name-service cache commands in advanced privilege, but it is not recommended to do so. For more detailed information, please see later sections in this document.
ONTAP also offers the additional benefit of using the caches while external name services are unavailable. If there is an entry in the cache, regardless if the entry’s TTL is expired or not, ONTAP will use that cache entry when external name services servers cannot be reached, thereby providing continued access to data served by the SVM.
Hosts Cache
There are two individual host caches; forward-lookup and reverse-lookup but the hosts cache settings are controlled as a whole. When a record is retrieved from DNS, the TTL of that record will be used for the cache TTL, otherwise, the default TTL in the host cache settings will be used (24 hours). The default for negative entries (host not found) is 60 seconds. Changing DNS settings does not affect the cache contents in any way.
The network ping command does not use the name services hosts cache when using a hostname.
User and Group Cache
The user and group caches consist of three categories; passwd (user), group and group membership.
Cluster RBAC access does not use the any of the caches
Passwd (User) Cache
User cache consists of two caches, passwd and passwd-by-uid. The caches only cache the name, uid and gid aspects of the user data to conserve space since the other data such as homedir and shell are irrelevant for NAS access. When an entry is placed in the passwd cache, the corresponding entry is created in the passwd-by-uid cache. By the same token, when an entry is deleted from one cache, the corresponding entry will be deleted from the other cache. If you have an environment where there are disjointed username to uid mappings, there is an option to disable this behavior.
Group Cache
Like the passwd cache, the group cache consists of two caches, group and group-by-gid. When an entry is placed in the group cache, the corresponding entry is created in the group-by-gid cache. By the same token, when an entry is deleted from one cache, the corresponding entry will be deleted from the other cache. The full group membership is not cached to conseve space and is not necessary for NAS data access, therefore only the group name and gid are cached. If you have an environment where there are disjointed group name to gid mappings, there is an option to disable this behavior.
Group Membership Cache
In file and NIS environments, there is no efficient way to gather a list of groups a particular user is a member of, so for these environments ONTAP has a group membership cache to provide these efficiencies. The group membership cache consists of a single cache and contains a list of groups a user is a member of.
Netgroup Cache
Beginning in ONTAP 9.3, the various netgroup caches have been consolidated into 2 caches; a netgroup.byhost and a netgroup.byname cache. The netgroup.byhost cache is the first cache consulted for the netgroups a host is a part of. Next, if this information is not available, then the query reverts to gathering the full netgroup members and comparing that to the host. If the information is not in the cache, then the same process is performed against the netgroup ns-switch sources. If a host requesting access via a netgroup is found via the netgroup membership lookup process, that ip-to-netgroup mapping is always added to the netgroup.byhost cache for faster future access. This also leads to needing a lower TTL for the members cache so that changes in netgroup membership can be reflected in the ONTAP caches within the TTL timeframe.
Viewing cache entries
Each of the above came service caches and be viewed. This can be used to confirm whether or not expected results are gotten from name services servers. Each cache has its own individual options that you can use to filter the results of the cache to find what you are looking for. In order to view the cache, the name-services cache <cache> <subcache> show command is used.
Caches are unique per vserver, so it is suggested to view caches on a per-vserver basis. Below are some examples of the caches and the options.
If there are no entries in a particular cache, the following message will be shown:
ontap9-tme-8040::*> name-service cache netgroups members show
(vserver services name-service cache netgroups members show)
This table is currently empty.
There you have it! New cache methodology in ONTAP 9.3. If you’re using NAS and name services in ONTAP, it’s highly recommended to go to ONTAP 9.3 to take advantage of this new feature.
Thoughts on Industrial Computing. Or sometimes economics or music or trail runnning or carpentry. Employee of Datto. Proud New Englander and Schaghticoke. Site is my own.
Why Is The Internet Broken? 