When I’m testing stuff out for customer deployments that I don’t work with a ton, I like to keep notes on the work so I can reference it later for TRs or other things. A blog is a great place to do that, as it might help other people in similar scenarios. This won’t be an exhaustive list, and it is certain to change over time and possibly make its way into a TR, but here we go…
ONTAP and Multiprotocol NAS
Before I get into the MacOS client stuff, we need to understand ONTAP multiprotocol NAS, as it can impact how MacOS clients behave.
In ONTAP, you can serve the same datasets to clients regardless of the NAS protocol they use (SMB or NFS). Many clients can actually do both protocols – MacOS is one of those clients.
The way ONTAP does Multiprotocol NAS (and keeps permissions predictable) is via name mappings and volume “security styles,” which controls what kind of ACLs are in use. TR-4887 goes into more detail on how all that works, but at a high level:
NTFS security styles use NTFS ACLs
SMB clients will map to UNIX users and NFS clients will require mappings to valid Windows users for authentication. Then, permissions are controlled via ACLs. Chmod/chown from NFS clients will fail.
UNIX security styles use UNIX mode bits (rwx) and/or NFSv4 ACLs
SMB clients will require mappings to a valid UNIX user for permissions; NFS clients will only require mapping to a UNIX user name if using NFSv4 ACLs. SMB clients can do *some* permissions changes, but on a very limited basis.
Mixed security styles always use either UNIX or NTFS effective security styles, based on last ACL change
Basically, if an NFS client chmods a file, it switches to UNIX security style. If an SMB client changes ownership of the file, it flips back to NTFS security style. This allows you to change permissions from any client, but you need to ensure you have proper name mappings in place to avoid undesired permission behavior. Generally, we recommend avoiding mixed security styles in most cases.
MacOS NFS Client Considerations
When using MacOS for an NFS client, there are a few things I’ve run into the past week or two while testing that you would want to know to avoid issues.
MacOS can be configured to use Active Directory LDAP for UNIX Identities
When you’re doing multiprotocol NAS (even if the clients will only do NFS, your volumes might have NTFS style permissions), you want to try to use a centralized name service like LDAP so that ONTAP, SMB clients and NFS clients all agree on who the users are, what groups they belong to, what numeric IDs they have, etc. If ONTAP thinks a user has a numeric ID of 1234 and the client things that user has a numeric ID of 5678, then you likely won’t get the access you expected. I wrote up a blog on configuring MacOS clients to use AD LDAP here:
MacOS clients can also be configured to use single sign on with AD and NFS home directories
Your MacOS clients – once added to AD in the blog post above – can now log in using AD accounts. There’s also an additional tab in the Directory Utility that allows you to auto-create home directories when a new user logs in to the MacOS client.
But you can also configure the auto-created home directories to leverage an NFS mount on the ONTAP storage system. You can configure the MacOS client to automount homedirs and then configure the MacOS client to use that path. (This process varies based on Mac version; I’m on 10.14.4 Catalina)
By default, the homedir path is /home in auto_master. We can use that.
Then, chmod the /etc/auto_home file to 644:
$ sudo chmod 644 /etc/auto_home
Create a volume on the ONTAP cluster for the homedirs and ensure it’s able to be mounted from the MacOS clients via the export policy rules (TR-4067 covers export policy rules):
::*> vol show -vserver DEMO -volume machomedirs -fields junction-path,policy vserver volume policy junction-path ------- ----------- ------- ---------------- DEMO machomedirs default /machomedirs
Create qtrees for each user and set the user/group and desired UNIX permissions:
qtree create -vserver DEMO -volume machomedirs -qtree prof1 -user prof1 -group ProfGroup -unix-permissions 755 qtree create -vserver DEMO -volume machomedirs -qtree student1 -user student1 -group group1 -unix-permissions 755
(For best results on Mac clients, use UNIX security styles.)
Then modify the automount /etc/auto_home file to use that path for homedir mounts. When a user logs in, the homedir will auto mount.
This is the line I used:
* -fstype=nfs nfs://demo:/machomedirs/&
And I also add the home mount
Then apply the automount change:
$ sudo automount -cv
automount: /net updated
automount: /home updated
automount: /Network/Servers updated
automount: no unmounts
Now, when I cd to /home/username, it automounts that path:
$ cd /home/prof1
$ mount
demo:/machomedirs/prof1 on /home/prof1 (nfs, nodev, nosuid, automounted, nobrowse)
But if I want that path to be the new homedir path, I would need to log in as that user and then go to “System Preferences -> Users and Groups” and right click the user. Then select “Advanced Options.”
Then you’d need to restart. Once that happens, log in again and when you first open Terminal, it will use the NFS homedir path.
NOTE: You may want to test if the Mac client can manually mount the homedir before testing logins. If the client can’t automount the homedir on login things will break.
Alternately, you can create a user with the same name as the AD account and then modify the homedir path (this removes the need to login). The Mac will pick up the correct UID, but the group ID may need to be changed.
If you use SMB shares for your home directories, it’s as easy as selecting “Use UNC path” in the User Experience area of Directory Utility (there’s no way to specify NFS here):
With new logins, the profile will get created in the qtree you created for the homedir (and you’ll go through the typical initial Mac setup screens):
# ls -la
total 28
drwxrwxr-x 6 student1 group1 4096 Apr 14 16:39 .
drwxr-xr-x 6 root root 4096 Apr 14 15:28 ..
drwx------ 2 student1 group1 4096 Apr 14 16:39 Desktop
drwx------ 2 student1 group1 4096 Apr 14 16:35 Downloads
drwxr-xr-x 25 student1 group1 4096 Apr 14 16:39 Library
-rw-r--r-- 1 student1 group1 4096 Apr 14 16:35 ._Library
drwx------ 4 student1 group1 4096 Apr 14 16:35 .Spotlight-V100
When you open terminal, it automounts the NFS home directory mount for that user and drops you right into your folder!
Mac NFS Considerations, Caveats, Issues
If you’re using NFS on Mac clients, there are two main things to remember:
- Volumes/qtrees using UNIX security styles work best with NFS in general
- Terminal/CLI works better than Finder in nearly all instances
If you have to/want to use Finder, or you have to/want to use NTFS security styles for multiprotocol, then there are some things you’d want to keep in mind.
- If possible, connect the Mac client to the Active Directory domain and use LDAP for UNIX identities as described above.
- Ensure your users/groups are all resolving properly on the Mac clients and ONTAP system. TR-4887 and TR-4835 cover some commands you can use to check users and groups, name mappings, group memberships, etc.
- If you’re using NTFS security style volumes/qtrees and want the Finder to work properly for copies to and from the NFS mount, configure the NFS export policy rule to set -ntfs-unix-security-ops to “ignore” – Finder will bail out if ONTAP returns an error, so we want to silently fail those operations (such as SETATTR; see below).
- When you open a file for reading/writing (such as a text file), Mac creates a ._filename file along with it. Depending on how many files you have in your volume, this can be an issue. For example, if you open 1 million files and Mac creates 1 million corresponding ._filename files, that starts to add up. Don’t worry! You’re not alone: https://apple.stackexchange.com/questions/14980/why-are-dot-underscore-files-created-and-how-can-i-avoid-them
- If you’re using DFS symlinks, check out this KB: DFS links do not work on MAC OS client, with ONTAP 9.5 and symlinks enabled
I’ve also run into some interesting behaviors with Mac/Finder/SMB and junction paths in ONTAP, as covered in this blog:
Workaround for Mac Finder errors when unzipping files in ONTAP
One issue that I did a pretty considerable amount of analysis on was the aforementioned “can’t copy using Finder.” Here are the dirty details…
Permissions Error When Copying a File to a NFS Mount in ONTAP using Finder
In this case, a file copy worked using Terminal, but was failing with permissions errors when using Finder and complaining about the file already existing.
First, it wants a login (which shouldn’t be needed):
Then it says this:
If you select “Replace” this is the error:
If you select “Stop” it stops and you are left with an empty 0 byte “file” – so the copy failed.
If you select “Keep Both” the Finder goes into an infinite loop of 0 byte file creations. I stopped mine at around 2500 files (forced an unmount):
# ls -al | wc -l 1981 # ls -al | wc -l 2004 # ls -al | wc -l 2525
So what does that happen? Well, in a packet trace, I saw the following:
The SETATTR fails on CREATE (expected in NFS operations on NTFS security style volumes in ONTAP, but not expected for NFS clients as per RFC standards):
181 60.900209 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 182), DH: 0x8ec2d57b/copy-file-finder << Mac NFS client checks if the file exists 182 60.900558 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 181) Error: NFS3ERR_NOENT << does not exist, so let’s create it! 183 60.900633 x.x.x.x x.x.x.y NFS 238 V3 CREATE Call (Reply In 184), DH: 0x8ec2d57b/copy-file-finder Mode: EXCLUSIVE << creates the file 184 60.901179 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 183) 185 60.901224 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 186), FH: 0x7b82dffd 186 60.901564 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 185) Error: NFS3ERR_PERM << fails setting attributes, which also fails the copy of the actual file data, so we have a 0 byte file
Then it REMOVES the file (since the initial operation fails) and creates it again, and SETATTR fails again. This is where that “Keep Both” loop behavior takes place.
229 66.995698 x.x.x.x x.x.x.y NFS 210 V3 REMOVE Call (Reply In 230), DH: 0x8ec2d57b/copy-file-finder
233 67.006816 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 234), DH: 0x8ec2d57b/copy-file-finder
234 67.007166 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 233) Error: NFS3ERR_NOENT
247 67.036056 x.x.x.x x.x.x.y NFS 238 V3 CREATE Call (Reply In 248), DH: 0x8ec2d57b/copy-file-finder Mode: EXCLUSIVE
248 67.037662 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 247)
249 67.037732 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 250), FH: 0xc33bff48
250 67.038534 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 249) Error: NFS3ERR_PERM
With Terminal, it operates a little differently. Rather than bailing out after the SETATTR failure, it just retries it:
11 19.954145 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 12), DH: 0x8ec2d57b/copy-file-finder 12 19.954496 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 11) Error: NFS3ERR_NOENT 13 19.954560 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 14), DH: 0x8ec2d57b/copy-file-finder 14 19.954870 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 13) Error: NFS3ERR_NOENT 15 19.954930 x.x.x.x x.x.x.y NFS 258 V3 CREATE Call (Reply In 18), DH: 0x8ec2d57b/copy-file-finder Mode: UNCHECKED 16 19.954931 x.x.x.x x.x.x.y NFS 230 V3 LOOKUP Call (Reply In 17), DH: 0x8ec2d57b/._copy-file-finder 17 19.955497 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 16) Error: NFS3ERR_NOENT 18 19.957114 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 15) 25 19.959031 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 26), FH: 0x8bcb16f1 26 19.959512 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 25) Error: NFS3ERR_PERM 27 19.959796 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 28), FH: 0x8bcb16f1 << Hey let's try again and ask in a different way! 28 19.960321 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 27)
The first SETATTR tries to chmod to 700:
Mode: 0700, S_IRUSR, S_IWUSR, S_IXUSR
The retry uses 777. Since the file already shows as 777, it succeeds (because it was basically fooled):
Mode: 0777, S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTH
Since Finder bails on the error, setting the NFS server to return no error here for this export (ntfs-unix-security-ops ignore) on this client allows the copy to succeed. You can create granular rules in your export policy rules to just set that option for your Mac clients.
Now, why do our files all show as 777?
Displaying NTFS Permissions via NFS
Because NFS doesn’t understand NTFS permissions, the job to translate user identities into valid access rights falls onto the shoulders of ONTAP. A UNIX user maps to a Windows user and then that Windows user is evaluated against the folder/file ACLs.
So “777” here doesn’t mean we have wide open access; we only have access based on the Windows ACL. Instead, it just means “the Linux client can’t view the access level for that user.” In most cases, this isn’t a huge problem. But sometimes, you need files/folders not to show 777 (like for applications that don’t allow 777).
In that case, you can control somewhat how NFS clients display NTFS ACLs in “ls” commands with the NFS server option ntacl-display-permissive-perms.
[-ntacl-display-permissive-perms {enabled|disabled}] - Display maximum NT ACL Permissions to NFS Client (privilege: advanced)
This optional parameter controls the permissions that are displayed to NFSv3 and NFSv4 clients on a file or directory that has an NT ACL set. When true, the displayed permissions are based on the maximum access granted by the NT ACL to any user. When false, the displayed permissions are based on the minimum access granted by the NT ACL to any user. The default setting is false.The default setting of “false” is actually “disabled.” When that option is enabled, this is the file/folder view:
When that option is disabled (the default):
This option is covered in more detail in TR-4067, but it doesn’t require a remount to take effect. It may take some time for the access caches to clear to see the results, however.
Keep in mind that these listings are approximations of the access as seen by the current user. If the option is disabled, you see the minimum access; if the option is enabled, you see the maximum access. For example, the “test” folder above shows 555 when the option is disabled, but 777 when the option is enabled.
These are the actual permissions on that folder:
::*> vserver security file-directory show -vserver DEMO -path /FG2/test Vserver: DEMO File Path: /FG2/test File Inode Number: 10755 Security Style: ntfs Effective Style: ntfs DOS Attributes: 10 DOS Attributes in Text: ----D--- Expanded Dos Attributes: - UNIX User Id: 1102 UNIX Group Id: 10002 UNIX Mode Bits: 777 UNIX Mode Bits in Text: rwxrwxrwx ACLs: NTFS Security Descriptor Control:0x8504 Owner:BUILTIN\Administrators Group:NTAP\ProfGroup DACL - ACEs ALLOW-Everyone-0x1200a9-OI|CI (Inherited) ALLOW-NTAP\prof1-0x1f01ff-OI|CI (Inherited)
Here are the expanded ACLs:
Owner:BUILTIN\Administrators
Group:NTAP\ProfGroup
DACL - ACEs
ALLOW-Everyone-0x1200a9-OI|CI (Inherited)
0... .... .... .... .... .... .... .... = Generic Read
.0.. .... .... .... .... .... .... .... = Generic Write
..0. .... .... .... .... .... .... .... = Generic Execute
...0 .... .... .... .... .... .... .... = Generic All
.... ...0 .... .... .... .... .... .... = System Security
.... .... ...1 .... .... .... .... .... = Synchronize
.... .... .... 0... .... .... .... .... = Write Owner
.... .... .... .0.. .... .... .... .... = Write DAC
.... .... .... ..1. .... .... .... .... = Read Control
.... .... .... ...0 .... .... .... .... = Delete
.... .... .... .... .... ...0 .... .... = Write Attributes
.... .... .... .... .... .... 1... .... = Read Attributes
.... .... .... .... .... .... .0.. .... = Delete Child
.... .... .... .... .... .... ..1. .... = Execute
.... .... .... .... .... .... ...0 .... = Write EA
.... .... .... .... .... .... .... 1... = Read EA
.... .... .... .... .... .... .... .0.. = Append
.... .... .... .... .... .... .... ..0. = Write
.... .... .... .... .... .... .... ...1 = Read
ALLOW-NTAP\prof1-0x1f01ff-OI|CI (Inherited)
0... .... .... .... .... .... .... .... = Generic Read
.0.. .... .... .... .... .... .... .... = Generic Write
..0. .... .... .... .... .... .... .... = Generic Execute
...0 .... .... .... .... .... .... .... = Generic All
.... ...0 .... .... .... .... .... .... = System Security
.... .... ...1 .... .... .... .... .... = Synchronize
.... .... .... 1... .... .... .... .... = Write Owner
.... .... .... .1.. .... .... .... .... = Write DAC
.... .... .... ..1. .... .... .... .... = Read Control
.... .... .... ...1 .... .... .... .... = Delete
.... .... .... .... .... ...1 .... .... = Write Attributes
.... .... .... .... .... .... 1... .... = Read Attributes
.... .... .... .... .... .... .1.. .... = Delete Child
.... .... .... .... .... .... ..1. .... = Execute
.... .... .... .... .... .... ...1 .... = Write EA
.... .... .... .... .... .... .... 1... = Read EA
.... .... .... .... .... .... .... .1.. = Append
.... .... .... .... .... .... .... ..1. = Write
.... .... .... .... .... .... .... ...1 = ReadSo, prof1 has Full Control (7) and “Everyone” has Read (5). That’s where the minimum/maximum permissions show up. So you won’t get *exact* permissions here. If you want exact permission views, consider using UNIX security styles.
DS_Store files
Mac will leave these little files laying around as users browse shares. In a large environment, that can start to create clutter, so you may want to consider disabling the creation of these on network shares (such as NFS mounts), as per this:
http://hints.macworld.com/article.php?story=2005070300463515http://hints.macworld.com/article.php?story=2005070300463515
If you have questions, comments or know of some other weirdness in MacOS with NFS, comment below!
Why Is The Internet Broken? 
Hey Justin, interesting read.
The Mac OS nfs client… do you always use the native one? Or are you installing another variant?
Ollie
LikeLike
Just used the native one, as that’s what I see most people using.
LikeLike
Terrific thanks, good to hear it worked, we might look at leaving smb behind on the Mac.
LikeLike
Hey Justin,
Wanted to ask if you or any of your readers discoverd performance issues when using SMB or NFS on MAC OS. The workload is Adobe Premiere Pro and simple file copy via Finder for a video editing workflow.
Client is directly connected via single 10G Eth.
Performance analysis gave some hints that MAC OS is not generating enough pressure on the storage and we have not found software tweaks to become faster.
Would be great if someone has some insights for this workload.
regards,
Gerold
LikeLike
I’d suggest reaching out to Apple or NetApp support. But I know for SMB, ONTAP doesn’t yet support some of the Apple-specific extensions.
LikeLike