MacOS NFS Clients with ONTAP – Tips and Considerations

When I’m testing stuff out for customer deployments that I don’t work with a ton, I like to keep notes on the work so I can reference it later for TRs or other things. A blog is a great place to do that, as it might help other people in similar scenarios. This won’t be an exhaustive list, and it is certain to change over time and possibly make its way into a TR, but here we go…

ONTAP and Multiprotocol NAS

Before I get into the MacOS client stuff, we need to understand ONTAP multiprotocol NAS, as it can impact how MacOS clients behave.

In ONTAP, you can serve the same datasets to clients regardless of the NAS protocol they use (SMB or NFS). Many clients can actually do both protocols – MacOS is one of those clients.

The way ONTAP does Multiprotocol NAS (and keeps permissions predictable) is via name mappings and volume “security styles,” which controls what kind of ACLs are in use. TR-4887 goes into more detail on how all that works, but at a high level:

NTFS security styles use NTFS ACLs

SMB clients will map to UNIX users and NFS clients will require mappings to valid Windows users for authentication. Then, permissions are controlled via ACLs. Chmod/chown from NFS clients will fail.

UNIX security styles use UNIX mode bits (rwx) and/or NFSv4 ACLs

SMB clients will require mappings to a valid UNIX user for permissions; NFS clients will only require mapping to a UNIX user name if using NFSv4 ACLs. SMB clients can do *some* permissions changes, but on a very limited basis.

Mixed security styles always use either UNIX or NTFS effective security styles, based on last ACL change

Basically, if an NFS client chmods a file, it switches to UNIX security style. If an SMB client changes ownership of the file, it flips back to NTFS security style. This allows you to change permissions from any client, but you need to ensure you have proper name mappings in place to avoid undesired permission behavior. Generally, we recommend avoiding mixed security styles in most cases.

MacOS NFS Client Considerations

When using MacOS for an NFS client, there are a few things I’ve run into the past week or two while testing that you would want to know to avoid issues.

MacOS can be configured to use Active Directory LDAP for UNIX Identities

When you’re doing multiprotocol NAS (even if the clients will only do NFS, your volumes might have NTFS style permissions), you want to try to use a centralized name service like LDAP so that ONTAP, SMB clients and NFS clients all agree on who the users are, what groups they belong to, what numeric IDs they have, etc. If ONTAP thinks a user has a numeric ID of 1234 and the client things that user has a numeric ID of 5678, then you likely won’t get the access you expected. I wrote up a blog on configuring MacOS clients to use AD LDAP here:

MacOS clients can also be configured to use single sign on with AD and NFS home directories

Your MacOS clients – once added to AD in the blog post above – can now log in using AD accounts. There’s also an additional tab in the Directory Utility that allows you to auto-create home directories when a new user logs in to the MacOS client.

But you can also configure the auto-created home directories to leverage an NFS mount on the ONTAP storage system. You can configure the MacOS client to automount homedirs and then configure the MacOS client to use that path. (This process varies based on Mac version; I’m on 10.14.4 Catalina)

By default, the homedir path is /home in auto_master. We can use that.

Then, chmod the /etc/auto_home file to 644:

$ sudo chmod 644 /etc/auto_home

Create a volume on the ONTAP cluster for the homedirs and ensure it’s able to be mounted from the MacOS clients via the export policy rules (TR-4067 covers export policy rules):

::*> vol show -vserver DEMO -volume machomedirs -fields junction-path,policy
vserver volume      policy  junction-path
------- ----------- ------- ----------------
DEMO    machomedirs default /machomedirs

Create qtrees for each user and set the user/group and desired UNIX permissions:

qtree create -vserver DEMO -volume machomedirs -qtree prof1 -user prof1 -group ProfGroup -unix-permissions 755
qtree create -vserver DEMO -volume machomedirs -qtree student1 -user student1 -group group1 -unix-permissions 755

(For best results on Mac clients, use UNIX security styles.)

Then modify the automount /etc/auto_home file to use that path for homedir mounts. When a user logs in, the homedir will auto mount.

This is the line I used:

* -fstype=nfs nfs://demo:/machomedirs/&

And I also add the home mount

Then apply the automount change:

$ sudo automount -cv
automount: /net updated
automount: /home updated
automount: /Network/Servers updated
automount: no unmounts

Now, when I cd to /home/username, it automounts that path:

$ cd /home/prof1
$ mount
demo:/machomedirs/prof1 on /home/prof1 (nfs, nodev, nosuid, automounted, nobrowse)

But if I want that path to be the new homedir path, I would need to log in as that user and then go to “System Preferences -> Users and Groups” and right click the user. Then select “Advanced Options.”

Then you’d need to restart. Once that happens, log in again and when you first open Terminal, it will use the NFS homedir path.

NOTE: You may want to test if the Mac client can manually mount the homedir before testing logins. If the client can’t automount the homedir on login things will break.

Alternately, you can create a user with the same name as the AD account and then modify the homedir path (this removes the need to login). The Mac will pick up the correct UID, but the group ID may need to be changed.

If you use SMB shares for your home directories, it’s as easy as selecting “Use UNC path” in the User Experience area of Directory Utility (there’s no way to specify NFS here):

With new logins, the profile will get created in the qtree you created for the homedir (and you’ll go through the typical initial Mac setup screens):

# ls -la
total 28
drwxrwxr-x 6 student1 group1 4096 Apr 14 16:39 .
drwxr-xr-x 6 root root 4096 Apr 14 15:28 ..
drwx------ 2 student1 group1 4096 Apr 14 16:39 Desktop
drwx------ 2 student1 group1 4096 Apr 14 16:35 Downloads
drwxr-xr-x 25 student1 group1 4096 Apr 14 16:39 Library
-rw-r--r-- 1 student1 group1 4096 Apr 14 16:35 ._Library
drwx------ 4 student1 group1 4096 Apr 14 16:35 .Spotlight-V100

When you open terminal, it automounts the NFS home directory mount for that user and drops you right into your folder!

Mac NFS Considerations, Caveats, Issues

If you’re using NFS on Mac clients, there are two main things to remember:

  • Volumes/qtrees using UNIX security styles work best with NFS in general
  • Terminal/CLI works better than Finder in nearly all instances

If you have to/want to use Finder, or you have to/want to use NTFS security styles for multiprotocol, then there are some things you’d want to keep in mind.

  • If possible, connect the Mac client to the Active Directory domain and use LDAP for UNIX identities as described above.
  • Ensure your users/groups are all resolving properly on the Mac clients and ONTAP system. TR-4887 and TR-4835 cover some commands you can use to check users and groups, name mappings, group memberships, etc.
  • If you’re using NTFS security style volumes/qtrees and want the Finder to work properly for copies to and from the NFS mount, configure the NFS export policy rule to set -ntfs-unix-security-ops to “ignore” – Finder will bail out if ONTAP returns an error, so we want to silently fail those operations (such as SETATTR; see below).
  • When you open a file for reading/writing (such as a text file), Mac creates a ._filename file along with it. Depending on how many files you have in your volume, this can be an issue. For example, if you open 1 million files and Mac creates 1 million corresponding ._filename files, that starts to add up. Don’t worry! You’re not alone: https://apple.stackexchange.com/questions/14980/why-are-dot-underscore-files-created-and-how-can-i-avoid-them
  • If you’re using DFS symlinks, check out this KB: DFS links do not work on MAC OS client, with ONTAP 9.5 and symlinks enabled

I’ve also run into some interesting behaviors with Mac/Finder/SMB and junction paths in ONTAP, as covered in this blog:

Workaround for Mac Finder errors when unzipping files in ONTAP

One issue that I did a pretty considerable amount of analysis on was the aforementioned “can’t copy using Finder.” Here are the dirty details…

Permissions Error When Copying a File to a NFS Mount in ONTAP using Finder

In this case, a file copy worked using Terminal, but was failing with permissions errors when using Finder and complaining about the file already existing.

First, it wants a login (which shouldn’t be needed):

Then it says this:

If you select “Replace” this is the error:

If you select “Stop” it stops and you are left with an empty 0 byte “file” – so the copy failed.

If you select “Keep Both” the Finder goes into an infinite loop of 0 byte file creations. I stopped mine at around 2500 files (forced an unmount):

# ls -al | wc -l
1981
# ls -al | wc -l
2004
# ls -al | wc -l
2525

So what does that happen? Well, in a packet trace, I saw the following:

The SETATTR fails on CREATE (expected in NFS operations on NTFS security style volumes in ONTAP, but not expected for NFS clients as per RFC standards):

181  60.900209  x.x.x.x    x.x.x.y      NFS  226  V3 LOOKUP Call (Reply In 182), DH: 0x8ec2d57b/copy-file-finder << Mac NFS client checks if the file exists
182  60.900558  x.x.x.y   x.x.x.x      NFS  186  V3 LOOKUP Reply (Call In 181) Error: NFS3ERR_NOENT << does not exist, so let’s create it!
183  60.900633  x.x.x.x    x.x.x.y      NFS  238  V3 CREATE Call (Reply In 184), DH: 0x8ec2d57b/copy-file-finder Mode: EXCLUSIVE << creates the file
184  60.901179  x.x.x.y   x.x.x.x      NFS  362  V3 CREATE Reply (Call In 183)
185  60.901224  x.x.x.x    x.x.x.y      NFS  238  V3 SETATTR Call (Reply In 186), FH: 0x7b82dffd
186  60.901564  x.x.x.y   x.x.x.x      NFS  214  V3 SETATTR Reply (Call In 185) Error: NFS3ERR_PERM << fails setting attributes, which also fails the copy of the actual file data, so we have a 0 byte file

Then it REMOVES the file (since the initial operation fails) and creates it again, and SETATTR fails again. This is where that “Keep Both” loop behavior takes place.

229 66.995698 x.x.x.x x.x.x.y NFS 210 V3 REMOVE Call (Reply In 230), DH: 0x8ec2d57b/copy-file-finder
233 67.006816 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 234), DH: 0x8ec2d57b/copy-file-finder
234 67.007166 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 233) Error: NFS3ERR_NOENT
247 67.036056 x.x.x.x x.x.x.y NFS 238 V3 CREATE Call (Reply In 248), DH: 0x8ec2d57b/copy-file-finder Mode: EXCLUSIVE
248 67.037662 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 247)
249 67.037732 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 250), FH: 0xc33bff48
250 67.038534 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 249) Error: NFS3ERR_PERM

With Terminal, it operates a little differently. Rather than bailing out after the SETATTR failure, it just retries it:

11 19.954145 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 12), DH: 0x8ec2d57b/copy-file-finder
12 19.954496 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 11) Error: NFS3ERR_NOENT
13 19.954560 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 14), DH: 0x8ec2d57b/copy-file-finder
14 19.954870 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 13) Error: NFS3ERR_NOENT
15 19.954930 x.x.x.x x.x.x.y NFS 258 V3 CREATE Call (Reply In 18), DH: 0x8ec2d57b/copy-file-finder Mode: UNCHECKED
16 19.954931 x.x.x.x x.x.x.y NFS 230 V3 LOOKUP Call (Reply In 17), DH: 0x8ec2d57b/._copy-file-finder
17 19.955497 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 16) Error: NFS3ERR_NOENT
18 19.957114 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 15)
25 19.959031 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 26), FH: 0x8bcb16f1
26 19.959512 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 25) Error: NFS3ERR_PERM
27 19.959796 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 28), FH: 0x8bcb16f1 << Hey let's try again and ask in a different way!
28 19.960321 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 27)

The first SETATTR tries to chmod to 700:

Mode: 0700, S_IRUSR, S_IWUSR, S_IXUSR

The retry uses 777. Since the file already shows as 777, it succeeds (because it was basically fooled):

Mode: 0777, S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTH

Since Finder bails on the error, setting the NFS server to return no error here for this export (ntfs-unix-security-ops ignore) on this client allows the copy to succeed. You can create granular rules in your export policy rules to just set that option for your Mac clients.

Now, why do our files all show as 777?

Displaying NTFS Permissions via NFS

Because NFS doesn’t understand NTFS permissions, the job to translate user identities into valid access rights falls onto the shoulders of ONTAP. A UNIX user maps to a Windows user and then that Windows user is evaluated against the folder/file ACLs.

So “777” here doesn’t mean we have wide open access; we only have access based on the Windows ACL. Instead, it just means “the Linux client can’t view the access level for that user.” In most cases, this isn’t a huge problem. But sometimes, you need files/folders not to show 777 (like for applications that don’t allow 777).

In that case, you can control somewhat how NFS clients display NTFS ACLs in “ls” commands with the NFS server option ntacl-display-permissive-perms.

[-ntacl-display-permissive-perms {enabled|disabled}] - Display maximum NT ACL Permissions to NFS Client (privilege: advanced)
This optional parameter controls the permissions that are displayed to NFSv3 and NFSv4 clients on a file or directory that has an NT ACL set. When true, the displayed permissions are based on the maximum access granted by the NT ACL to any user. When false, the displayed permissions are based on the minimum access granted by the NT ACL to any user. The default setting is false.

The default setting of “false” is actually “disabled.” When that option is enabled, this is the file/folder view:

When that option is disabled (the default):

This option is covered in more detail in TR-4067, but it doesn’t require a remount to take effect. It may take some time for the access caches to clear to see the results, however.

Keep in mind that these listings are approximations of the access as seen by the current user. If the option is disabled, you see the minimum access; if the option is enabled, you see the maximum access. For example, the “test” folder above shows 555 when the option is disabled, but 777 when the option is enabled.

These are the actual permissions on that folder:

::*> vserver security file-directory show -vserver DEMO -path /FG2/test
Vserver: DEMO
File Path: /FG2/test
File Inode Number: 10755
Security Style: ntfs
Effective Style: ntfs
DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
UNIX User Id: 1102
UNIX Group Id: 10002
UNIX Mode Bits: 777
UNIX Mode Bits in Text: rwxrwxrwx
ACLs: NTFS Security Descriptor
Control:0x8504
Owner:BUILTIN\Administrators
Group:NTAP\ProfGroup
DACL - ACEs
ALLOW-Everyone-0x1200a9-OI|CI (Inherited)
ALLOW-NTAP\prof1-0x1f01ff-OI|CI (Inherited)

Here are the expanded ACLs:

                     Owner:BUILTIN\Administrators
                     Group:NTAP\ProfGroup
                     DACL - ACEs
                       ALLOW-Everyone-0x1200a9-OI|CI (Inherited)
                          0... .... .... .... .... .... .... .... = Generic Read
                          .0.. .... .... .... .... .... .... .... = Generic Write
                          ..0. .... .... .... .... .... .... .... = Generic Execute
                          ...0 .... .... .... .... .... .... .... = Generic All
                          .... ...0 .... .... .... .... .... .... = System Security
                          .... .... ...1 .... .... .... .... .... = Synchronize
                          .... .... .... 0... .... .... .... .... = Write Owner
                          .... .... .... .0.. .... .... .... .... = Write DAC
                          .... .... .... ..1. .... .... .... .... = Read Control
                          .... .... .... ...0 .... .... .... .... = Delete
                          .... .... .... .... .... ...0 .... .... = Write Attributes
                          .... .... .... .... .... .... 1... .... = Read Attributes
                          .... .... .... .... .... .... .0.. .... = Delete Child
                          .... .... .... .... .... .... ..1. .... = Execute
                          .... .... .... .... .... .... ...0 .... = Write EA
                          .... .... .... .... .... .... .... 1... = Read EA
                          .... .... .... .... .... .... .... .0.. = Append
                          .... .... .... .... .... .... .... ..0. = Write
                          .... .... .... .... .... .... .... ...1 = Read

                       ALLOW-NTAP\prof1-0x1f01ff-OI|CI (Inherited)
                          0... .... .... .... .... .... .... .... = Generic Read
                          .0.. .... .... .... .... .... .... .... = Generic Write
                          ..0. .... .... .... .... .... .... .... = Generic Execute
                          ...0 .... .... .... .... .... .... .... = Generic All
                          .... ...0 .... .... .... .... .... .... = System Security
                          .... .... ...1 .... .... .... .... .... = Synchronize
                          .... .... .... 1... .... .... .... .... = Write Owner
                          .... .... .... .1.. .... .... .... .... = Write DAC
                          .... .... .... ..1. .... .... .... .... = Read Control
                          .... .... .... ...1 .... .... .... .... = Delete
                          .... .... .... .... .... ...1 .... .... = Write Attributes
                          .... .... .... .... .... .... 1... .... = Read Attributes
                          .... .... .... .... .... .... .1.. .... = Delete Child
                          .... .... .... .... .... .... ..1. .... = Execute
                          .... .... .... .... .... .... ...1 .... = Write EA
                          .... .... .... .... .... .... .... 1... = Read EA
                          .... .... .... .... .... .... .... .1.. = Append
                          .... .... .... .... .... .... .... ..1. = Write
                          .... .... .... .... .... .... .... ...1 = Read

So, prof1 has Full Control (7) and “Everyone” has Read (5). That’s where the minimum/maximum permissions show up. So you won’t get *exact* permissions here. If you want exact permission views, consider using UNIX security styles.

DS_Store files

Mac will leave these little files laying around as users browse shares. In a large environment, that can start to create clutter, so you may want to consider disabling the creation of these on network shares (such as NFS mounts), as per this:

http://hints.macworld.com/article.php?story=2005070300463515http://hints.macworld.com/article.php?story=2005070300463515

If you have questions, comments or know of some other weirdness in MacOS with NFS, comment below!

How to Configure MacOS to Use Active Directory LDAP for UNIX users/groups

In NetApp ONTAP, it’s possible to serve data to NAS clients over SMB and NFS, including the same datasets. This is known as “multiprotocol NAS” and I cover the best practices for that in the new TR-4887:

TR-4887: Multiprotocol NAS Best Practices in ONTAP

When you do multiprotocol NAS in ONTAP (or really, and storage system), it’s usually best to leverage a centralized repository for user names, group names and numeric IDs that the NAS clients and NAS servers all point to (such as LDAP). That way, you get no surprises when accessing files and folders – user and groups get the expected ownership and permissions.

I cover LDAP in ONTAP in TR-4835:

TR-4835: LDAP in NetApp ONTAP

One of the more commonly implemented solutions for LDAP in environments that serve NFS and SMB is Active Directory. In these environments, you can either use either native UNIX attributes or a 3rd party utility, such as Centrify. (Centrify basically just uses the AD UNIX attributes and centralizes the management into a GUI – both are covered in the LDAP TR.)

While most Linux clients are fairly straightforward for LDAP integration, MacOS does things slightly differently. However, it’s pretty easy to configure.

Note: The steps may vary based on your environment configs and this covers just AD LDAP; not OpenLDAP/IDM or other Linux-based LDAP.

macOS Big Sur is here - Apple

Step 1: Ensure the Mac is configured to use the same DNS as the AD domain

This is done via “Network” settings in System Preferences. DNS is important here because it will be used to query the AD domain when we bind the Mac for the Directory Services. In the following, I’ve set the DNS server IP and the search domain to my AD domain “NTAP.LOCAL”:

Then I tested the domain lookup in Terminal:

Step 2: Configure Directory Services to use Active Directory

The “Directory Utility” is what we’ll use here. It’s easiest to use the spotlight search to find it.

Essentially, this process adds the MacOS to the Active Directory domain (as you would with a Windows server or a Linux box with “realm join”) and then configures the LDAP client on the Mac to leverage specific attributes for LDAP queries.

In the above, I’ve used uidNumber and gidNumber as the attributes. You can also control/view these via the CLI command “dsconfigad”:

Configure domain access in Directory Utility on Mac

I can see in my Windows AD domain the machine account was created:

A few caveats here about the default behavior for this:

  • LDAP queries will be encrypted by default, so if you’re trying to troubleshoot via packet capture, you won’t see a ton of useful info (such as attributes used for queries). To disable this (mainly for troubleshooting purposes):
$ dsconfigad -packetsign disable -packetencrypt disable
  • MacOS uses sAMAccountName as the user name/uid value, so it should work fine with AD out of the gate
  • MacOS adds additional Mac-specific system groups to the “id” output (such as netaccounts:62, and GIDs 701/702); these may need to be added to LDAP, depending on file ownership
  • LDAP queries to AD from Mac will use the Global Catalog port 3268 by default when using Active Directory config (which I was able to see from a packet capture)

This use of the global catalog port can be problematic, as standard LDAP configurations in AD don’t set the Global Catalog to replicate attributes, but rather uses the standard port 389/636 for LDAP communication. AD doesn’t replicate the UNIX attributes across the global catalog by default for LDAP, so you’d have to configure that manually (covered in TR-4835) or modify the port the Mac uses for LDAP.

My AD domain does have the attributes that replicate via the Global Catalog, so the LDAP lookups work for me from the Mac:

Here’s what the prof1 user looks like from a CentOS client:

# id prof1
uid=1102(prof1) gid=10002(ProfGroup) groups=10002(ProfGroup),10000(Domain Users),48(apache-group),1101(group1),1202(group2),1203(group3)

This is how that user looks from the ONTAP CLI:

cluster::> set advanced; access-check authentication show-creds -node node1 -vserver DEMO -unix-user-name prof1 -list-name true -list-id true
UNIX UID: 1102 (prof1) <> Windows User: S-1-5-21-3552729481-4032800560-2279794651-1110 (NTAP\prof1 (Windows Domain User))
GID: 10002 (ProfGroup)
Supplementary GIDs:
10002 (ProfGroup)
10000 (Domain Users)
1101 (group1)
1202 (group2)
1203 (group3)
48 (apache-group)
Primary Group SID: S-1-5-21-3552729481-4032800560-2279794651-1111 NTAP\ProfGroup (Windows Domain group)
Windows Membership:
S-1-5-21-3552729481-4032800560-2279794651-1301 NTAP\apache-group (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1106 NTAP\group2 (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-513 NTAP\DomainUsers (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1105 NTAP\group1 (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1107 NTAP\group3 (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1111 NTAP\ProfGroup (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1231 NTAP\local-group.ntap (Windows Alias)
S-1-18-2 Service asserted identity (Windows Well known group)
S-1-5-32-551 BUILTIN\Backup Operators (Windows Alias)
S-1-5-32-544 BUILTIN\Administrators (Windows Alias)
S-1-5-32-545 BUILTIN\Users (Windows Alias)
User is also a member of Everyone, Authenticated Users, and Network Users
Privileges (0x22b7):
SeBackupPrivilege
SeRestorePrivilege
SeTakeOwnershipPrivilege
SeSecurityPrivilege
SeChangeNotifyPrivilege

Most people aren’t going to want/be allowed to crack open ADSIEdit and modify schema attributes, so you’d want to change how MacOS queries LDAP to use port 389 or 636. I’m currently waiting on word of how to do that from Apple, so I’ll update when I get that info. If you are reading this and already know, feel free to add to the comments!

Step 3: Mount the NetApp NFS export and test multiprotocol access

NFS mounts to UNIX security style volumes are pretty straightforward, so we won’t cover that here. Where it gets tricky is when your ONTAP volumes are NTFS security style. When that’s the case, a UNIX -> Windows name mapping occurs when using NFS, as we need to make sure the user trying to access the NTFS permissions truly has access.

This is the basic process:

  • MacOS NFS client sends a numeric UID and GID to the NetApp ONTAP system (if NFSv3 is used)
  • If the volume is NTFS security, ONTAP will try to translate the numeric IDs into user names. The method depends on the cluster config; in this case, we’ll use LDAP.
  • If the numeric IDs map to user names/group names, then ONTAP uses those UNIX names and tries to find a valid Windows name with the same names; if none exist, ONTAP looks for explicit name mapping rules and a default Windows user; if none of those work then access is denied.

I have mounted a volume to my MacOS client that uses NTFS security style.

This is the volume in ONTAP:

::*> vol show -vserver DEMO -volume FG2 -fields security-style
vserver volume security-style
------- ------ --------------
DEMO       FG2           ntfs

MacOS user IDs start at the ID 501; So my “admin” user ID is 501. This user doesn’t exist in LDAP.

ONTAP has a local user named “Podcast” but no valid Windows user mapping:

::*> set advanced; access-check authentication show-creds -node ontap9-tme-8040-01 -vserver DEMO -uid 501
(vserver services access-check authentication show-creds)
Vserver: DEMO (internal ID: 10)
Error: Get user credentials procedure failed
[ 33] Determined UNIX id 501 is UNIX user 'Podcast'
[ 34] Using a cached connection to ntap.local
[ 36] Trying to map 'Podcast' to Windows user 'Podcast' using
implicit mapping
[ 36] Successfully connected to ip 10.193.67.236, port 445
using TCP
[ 46] Successfully authenticated with DC oneway.ntap.local
[ 49] Could not find Windows name 'Podcast'
[ 49] Unable to map 'Podcast'. No default Windows user defined.
**[ 49] FAILURE: Name mapping for UNIX user 'Podcast' failed. No
** mapping found
Error: command failed: Failed to get user credentials. Reason: "SecD Error: Name mapping does not exist".

In addition, MacOS disables root by default:

https://support.apple.com/en-us/HT204012

So when I try to access this mount, it will attempt to use UID 501 and translate it to a UNIX user and then to a Windows user. Since ONTAP can’t translate UID 501 to a valid Windows user, this will fail and we’ll see it in the event log of the ONTAP CLI.

Here’s the access failure:

Here’s the ONTAP error:

ERROR secd.nfsAuth.noNameMap: vserver (DEMO) Cannot map UNIX name to CIFS name. Error: Get user credentials procedure failed
[ 33] Determined UNIX id 501 is UNIX user 'Podcast'
[ 34] Using a cached connection to ntap.local
[ 36] Trying to map 'Podcast' to Windows user 'Podcast' using implicit mapping
[ 36] Successfully connected to ip 10.193.67.236, port 445 using TCP
[ 46] Successfully authenticated with DC oneway.ntap.local
[ 49] Could not find Windows name 'Podcast'
[ 49] Unable to map 'Podcast'. No default Windows user defined.
**[ 49] FAILURE: Name mapping for UNIX user 'Podcast' failed. No mapping found

When I “su” to a user that *does* have a valid Windows user (such as prof1), this works fine and I can touch a file and get the proper owner/group:

Note that in the above, we see “root:wheel” owned folders; just because root is disabled by default on MacOS doesn’t mean that MacOS isn’t aware of the user. Those folders were created on a separate NFS client.

Also, note in the above that the file shows 777 permissions; this is because those are the allowed permissions for the prof1 user on that file. The permissions are defined by Windows/NTFS. Here, they are set to “Everyone:Full Control” by way of file inheritance. These are the new permissions. Profgroup (with prof1 and studen1 as members) gets write access. Administrator gets “Full Control.” Group10 (with only student1 as a member) gets read access.

In ONTAP, you can also control the way NTFS security style files are viewed on NFS clients with the NFS server option -ntacl-display-permissive-perms. TR-4887 covers that option in more detail.

Prof1 access view after permissions change (write access):

Student1 access view after permissions change (read access only via group ACL):

Read works, but write does not (by design!)

Student2 access view (write access defined by membership in ProfGroup):

Newuser1 access view (not a member of any groups in the ACL):

Newuser1 can create a new file, however, and it shows the proper owner. The permissions are 777 because of the inherited NTFS ACLs from the share:

As you can see, we will get the expected access for users and groups on Mac NFS using NTFS security styles, but the expected *views* won’t always line up. This is because there isn’t a direct correlation between NTFS and UNIX permissions, so we deliver an approximation. ONTAP doesn’t store ACLs for both NTFS and UNIX on disk; it only chooses one or the other. If you require exact NFS permission translation via Mac NFS, consider using UNIX security style and mode bits.

Addendum: Squashing root

In the event your MacOS users enable the root account and become the “root” user on the client, you can squash the root user to an anonymous user by using ONTAP’s export policies and rules. TR-4067 covers how to do this:

NFS Best Practices in ONTAP

Let me know if you have questions!

Workaround for Mac Finder errors when unzipping files in ONTAP

ONTAP allows you to mount volumes to other volumes in a Storage Virtual Machine, which provides a way for storage administrators to create their own folder structures across multiple nodes in a cluster. This is useful when you want to ensure the workload gets spread across nodes, but you can’t use FlexGroup volumes for whatever reason.

This graphic shows how that can work:

junctioned-volumes.png

In NAS environments, a client will ask for a file or folder location and ONTAP will re-direct the traffic to wherever that object lives. This is supposed to be transparent to the client, provided they follow standard NAS deployment steps.

However, not all NAS clients are created equal. Sometimes, Linux serves up SMB and will do things differently than Microsoft does. Windows also will do NFS, but it doesn’t entirely follow the NFS RFCs. So, occasionally, ONTAP doesn’t expect how a client handles something and stuff breaks.

Mac Finder

If you’ve ever used a Mac, you’ll know that the Finder can do somethings a little differently than the Terminal does. In this particular issue, we’ll focus on how Finder unzips files (when you double-click the file) in volumes that are mounted to other volumes in ONTAP.

One of our customers hit this issue, and after poking around a little bit, I figured out how to workaround the issue.

Here’s what they were doing:

  • SMB to Mac clients
  • Shares at the parent FlexVol level (ie, /vol1
  • FlexVols mounted to other FlexVols several levels deep (ie, /vol1/vol2/vol3)

When files are unzipped after accessing a share at a higher level and then drilling down into other folders (which are actually FlexVols mounted to other FlexVols), then unzipping in Finder via double-click fails.

When the shares are mounted at the same level as the FlexVol where the unzip is attempted, unzip works. When the Terminal is used to unzip, it works.

However, when your users refuse to use/are unable to use the Terminal and you don’t want to create hundreds of shares just to work around one issue, it’s an untenable situation.

So, I decided to dig into the issue…

Reproducing the issue

The best way to troubleshoot problems is to set up a lab environment and try to recreate the problem. This allows you freedom to gather logs, packet traces, etc. without bothering your customer or end user. So, I brought my trusty old 2011 MacBook running OS Sierra and mounted the SMB share in question.

These are the volumes and their junction paths:

DEMO inodes /shared/inodes
DEMO shared /shared

This is the share:

 Vserver: DEMO
 Share: shared
 CIFS Server NetBIOS Name: DEMO
 Path: /shared
 Share Properties: oplocks
 browsable
 changenotify
 show-previous-versions
 Symlink Properties: symlinks
 File Mode Creation Mask: -
 Directory Mode Creation Mask: -
 Share Comment: -
 Share ACL: Everyone / Full Control
 File Attribute Cache Lifetime: -
 Volume Name: shared
 Offline Files: manual
 Vscan File-Operations Profile: standard
 Maximum Tree Connections on Share: 4294967295
 UNIX Group for File Create: -

I turned up debug logging on the cluster (engage NetApp Support if you want to do this), got a packet trace on the Mac and reproduced the issue right away. Lucky me!

finder-error

I also tried a 3rd party unzip utility (Stuffit Expander) and it unzipped fine. So this was definitely a Finder/ONTAP/NAS interaction problem, which allowed me to focus on that.

Packet traces showed that the Finder was attempting to look for a folder called “.TemporaryItems/folders.501/Cleanup At Startup” but couldn’t find it – and couldn’t create it, apparently either. But it would created folders named “BAH.XXXX” instead, and they wouldn’t get cleaned up.

So, I thought, why not manually create the folder path, since it wasn’t able to do it on its own?

You can do this through Terminal, or via Finder. Keep in mind that the path above has “folders.501” – 501 is my uid, so check your users uid on the Mac and make sure the folder path is created using that uid. If you have multiple users that access the share, you may need to create multiple folders.xxx in .TemporaryItems.

If you do it via Finder, you may want to enable hidden files. I learned how to do that via this article:

https://ianlunn.co.uk/articles/quickly-showhide-hidden-files-mac-os-x-mavericks/

So I did that and then I unmounted the share and re-mounted, to make sure there wasn’t any weird cache issue lingering. You can check CIFS/SMB sessions, versions, etc with the following command, if you want to make sure they are closed:

cluster::*> cifs session show -vserver DEMO -instance

Vserver: DEMO

Node: node1
 Session ID: 16043510722553971076
 Connection ID: 390771549
 Incoming Data LIF IP Address: 10.x.x.x
 Workstation IP Address: 10.x.x.x
 Authentication Mechanism: NTLMv2
 User Authenticated as: domain-user
 Windows User: NTAP\prof1
 UNIX User: prof1
 Open Shares: 1
 Open Files: 1
 Open Other: 0
 Connected Time: 7m 49s
 Idle Time: 6m 2s
 Protocol Version: SMB3
 Continuously Available: No
 Is Session Signed: true
 NetBIOS Name: -
 SMB Encryption Status: unencrypted
 Connection Count: 1

Once I reconnected with the newly created folder path, double-click unzip worked perfectly!

Check it out yourself:

Note: You *may* have to enable the option is-use-junctions-as-reparse-points-enabled on your CIFS server. I haven’t tested with it off and on thoroughly, but I saw some inconsistency when it was disabled. For the record, it’s on by default.

You can check with:

::*> cifs options show -fields is-use-junctions-as-reparse-points-enabled

Give it a try and let me know how it works for you in the comments!

Issues installing Docker on OS 10.10 or later?

Today I was trying to install the Docker Toolbox on my Mac. It failed. Because I was able to fix it and did not see any other articles on how to do this that specifically referenced this app or issue, I decided to write it up. Because, community!

docker_toolbox_banner_icon

The installation appeared to work fine, but once I clicked the “Docker Terminal” icon, the terminal would launch with the following message:

Docker Machine is not installed. Please re-run the Toolbox Installer and try again.

Docker Machine installs by default to the /usr/local/bin directory in OS X. And when I tried to change that location in the install package, I didn’t have any luck.

That directory is locked down pretty tight (700 permissions, my user as the owner).

drwx------  24 parisi  wheel  816 Mar 25 10:52 bin

When I tried to open the directory up a bit and re-install, it would have the same issue. And, when I tried to cd directly into that directory, it either threw a permission denied or silently failed, even though I had allowed access:

$ sudo chmod 766 bin

$ ls -la
total 0
drwxr-xr-x   5 root    wheel  170 Mar 25 10:50 .
drwxr-xr-x@ 10 root    wheel  340 May  8  2015 ..
drwxr-xr-x   3 root    wheel  102 Apr 20  2015 Qt
drwxrw-rw-  24 parisi  wheel  816 Mar 25 10:52 bin
drwxr-xr-x   3 root    wheel  102 Mar 25 10:50 share

$ cd bin
-bash: cd: bin: Permission denied

$ sudo cd bin

$ pwd
/usr/local

And Docker commands failed:

$ docker
-bash: docker: command not found

Color me stumped.

So I turned to Google and found an article on Homebrew installations failing, but nothing specifically on Docker failing. I used the Homebrew workaround found in this article and it fixed my issue.

Here are the commands I ran:

$ sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local

Essentially, the command above does a recursive (-R) chown on the /usr/local directories as the logged in user (via whoami).

Before the change, /usr/local looked like this:

drwxr-xr-x     6 root  wheel    204 Mar 25 10:56 local

After the change:

drwxr-xr-x     6 parisi  admin    204 Mar 25 10:56 local

After that, I could run Docker commands:

$ pwd
/Users/parisi

$ docker
Usage: docker [OPTIONS] COMMAND [arg...]
       docker [ --help | -v | --version ]
A self-sufficient runtime for containers.

Options:
  --config=~/.docker              Location of client config files
  -D, --debug                     Enable debug mode
  -H, --host=[]                   Daemon socket(s) to connect to
  -h, --help                      Print usage
  -l, --log-level=info            Set the logging level
  --tls                           Use TLS; implied by --tlsverify
  --tlscacert=~/.docker/ca.pem    Trust certs signed only by this CA
  --tlscert=~/.docker/cert.pem    Path to TLS certificate file
  --tlskey=~/.docker/key.pem      Path to TLS key file
  --tlsverify                     Use TLS and verify the remote
  -v, --version                   Print version information and quit

...

And the Docker terminal starts correctly:

Creating CA: /Users/parisi/.docker/machine/certs/ca.pem
Creating client certificate: /Users/parisi/.docker/machine/certs/cert.pem
Running pre-create checks...
Creating machine...
(default) Copying /Users/parisi/.docker/machine/cache/boot2docker.iso to /Users/parisi/.docker/machine/machines/default/boot2docker.iso...
(default) Creating VirtualBox VM...
(default) Creating SSH key...
(default) Starting the VM...
(default) Check network to re-create if needed...
(default) Found a new host-only adapter: "vboxnet0"
(default) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: /usr/local/bin/docker-machine env default

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/

docker is configured to use the default machine with IP X.X.X.X
For help getting started, check out the docs at https://docs.docker.com

If you’re interested on more detail on the issue, check out the Homebrew blog, as well as this on System Integrity Protector (SIP):

https://support.apple.com/en-us/HT204899

Hopefully this helps someone else.

Another option, pointed out to me on Twitter, is to use the native Docker apps (still in beta):

https://blog.docker.com/2016/03/docker-for-mac-windows-beta/

If interested, I’ve written a couple other blogs on Docker.

TECH::Using NFS with Docker – Where does it fit in?

TECH::Docker + CIFS/SMB? That’s unpossible!