Behind the Scenes Episode 289 – NetApp and Rubrik: Better Together

Welcome to the Episode 289, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, we discuss the latest news in the NetApp/Rubrik partnership and how Rubrik works with Ben Kendall (Alliances Technical Partner Manager, Americas at Rubrik, https://www.linkedin.com/in/ben-kendall-6436609/) and PF Guglielmi (Alliances Field CTO @pfguglielmi) of Rubrik and Chris Maino (Manager, Americas Solutions Architects, chris.maino@netapp.com) of NetApp.

For more information:

https://www.rubrik.com/en/partners/technology-partners/netapp

https://www.rubrik.com/en/company/newsroom/press-releases/21/rubrik-and-netapp-extend-partnership

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 289 – NetApp and Rubrik: Better Together – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

How to see data transfer on an ONTAP cluster network

ONTAP clusters utilize a backend cluster network to allow multiple HA pairs to communicate and provide more scale for performance and capacity. This is done by allowing you to nondisruptively add new nodes (and, as a result, capacity and compute) into a cluster. Data will be accessible regardless of where you connect in the cluster. You can scale up to 24 nodes for NAS-only clusters, while being able to mix different HA pair types in the same cluster if you choose to offer different service levels for storage (such as performance tiers, capacity tiers, etc).

Network interfaces that serve data to clients live on physical ports on nodes and are floating/virtual IP addresses that can move to/from any node in the cluster. File systems for NAS are defined by Storage Virtual Machines (SVMs) and volumes. The SVMs own the IP addresses you would use to access data.

When using NAS (CIFS/SMB/NFS) for data access, you can connect to a data interface in the SVM that lives on any node in the cluster, regardless of where the data volume resides. The following graphic shows how that happens.

When you access a NAS volume on a data interface on the same node as the data volume, ONTAP can “cheat” a little and directly interact with that volume without having to do extra work.

If that data interface is on a different node than where the volume resides, then the NAS packet gets packaged up as a proprietary protocol and shipped over the cluster network backend to the node where the volume lives. This volume/node relationship is stored in an internal database in ONTAP so we always have a map to find volumes quickly. Once the NAS packet arrives on the destination node, it gets unpackaged, processed and then the response to the client goes back out the way it came.

Traversing the cluster network has a bit of a latency cost, however, as the packaging/unpackaging/traversal takes some time (more time than a local request). This manifests into slightly less performance for those workloads. The impact of that performance hit is negligible in most environments, but for latency-sensitive applications, there might be some noticeable performance degradation.

There are protocol features that help mitigate the remote I/O that can occur in a cluster, such as SMB node referrals and pNFS, but in scenarios where you can’t use either of those (SMB node referrals didn’t use Kerberos in earlier Windows versions; pNFS needs NFSv4.1 and later), then you’re going to likely have remote cluster traffic. As mentioned, in most cases this isn’t an issue, but it may be useful to have an easy way to find out if an ONTAP cluster is doing remote/cluster traffic.

Cluster level – Statistics show-periodic

To get a cluster-wide view if there is remote traffic on the cluster, you can use the advanced priv command “statistics show-periodic.” This command gives a wealth of information by default, such as:

  • CPU average/busy
  • Total ops/NFS ops/CIFS ops
  • FlexCache ops
  • Total data recieved/sent (Data and cluster network throughput)
  • Data received/sent (Data throughput only)
  • Cluster received/sent (Cluster throughput only)
  • Cluster busy % (how busy the cluster network is)
  • Disk reads/writes
  • Packets sent/received

We also have options to limit the intervals, define SVMs/vservers, etc.

::*> statistics show-periodic ?
[[-object] ] *Object
[ -instance ] *Instance
[ -counter ] *Counter
[ -preset ] *Preset
[ -node ] *Node
[ -vserver ] *Vserver
[ -interval ] *Interval in Seconds (default: 2)
[ -iterations ] *Number of Iterations (default: 0)
[ -summary {true|false} ] *Print Summary (default: true)
[ -filter ] *Filter Data

But for backend cluster traffic, we only care about a few of those, so we can filter the iterations for only what we want to view. In this case, I just want to look at the data sent/received and the cluster busy %.

::*> statistics show-periodic -counter total-recv|total-sent|data-recv|data-sent|cluster-recv|cluster-sent|cluster-busy

When I do that, I get a cleaner, easier to read capture. This is what it looks like when we have remote traffic. This is an NFSv4.1 workload without pNFS, using a mount wsize of 64K.

cluster1: cluster.cluster: 5/11/2021 14:01:49
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
    157MB   4.85MB    148MB   3.46MB      0%   8.76MB   1.39MB
    241MB   70.2MB    197MB   4.68MB      1%   43.1MB   65.5MB
    269MB    111MB    191MB   4.41MB      4%   78.1MB    107MB
    329MB   92.5MB    196MB   4.52MB      4%    133MB   88.0MB
    357MB    117MB    246MB   5.68MB      2%    111MB    111MB
    217MB   27.1MB    197MB   4.55MB      1%   20.3MB   22.5MB
    287MB   30.4MB    258MB   5.91MB      1%   28.7MB   24.5MB
    205MB   28.1MB    176MB   4.03MB      1%   28.9MB   24.1MB
cluster1: cluster.cluster: 5/11/2021 14:01:57
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
Minimums:
    157MB   4.85MB    148MB   3.46MB      0%   8.76MB   1.39MB
Averages for 8 samples:
    258MB   60.3MB    201MB   4.66MB      1%   56.5MB   55.7MB
Maximums:
    357MB    117MB    258MB   5.91MB      4%    133MB    111MB

As we can see, there is an average of 55.7MB sent and 56.5MB received over the cluster network each second; this accounts for an average of 1% of the available bandwidth, which means we have plenty of cluster network utilization left over.

When we look at the latency for this workload, this is what we see. (Using qos statistics latency show)

Policy Group            Latency
-------------------- ----------
-total-                364.00us
extreme-fixed          364.00us
-total-                619.00us
extreme-fixed          619.00us
-total-                490.00us
extreme-fixed          490.00us
-total-                409.00us
extreme-fixed          409.00us
-total-                422.00us
extreme-fixed          422.00us
-total-                474.00us
extreme-fixed          474.00us
-total-                412.00us
extreme-fixed          412.00us
-total-                372.00us
extreme-fixed          372.00us
-total-                475.00us
extreme-fixed          475.00us
-total-                436.00us
extreme-fixed          436.00us
-total-                474.00us
extreme-fixed          474.00us

This is what the cluster network looks like when I use pNFS for data locality:

cluster1: cluster.cluster: 5/11/2021 14:18:19
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
    208MB   6.24MB    206MB   4.76MB      0%   1.56MB   1.47MB
    214MB   5.37MB    213MB   4.85MB      0%    555KB    538KB
    214MB   6.27MB    213MB   4.80MB      0%   1.46MB   1.47MB
    219MB   5.95MB    219MB   5.40MB      0%    572KB    560KB
    318MB   8.91MB    317MB   7.44MB      0%   1.46MB   1.47MB
    203MB   5.16MB    203MB   4.62MB      0%    560KB    548KB
    205MB   6.09MB    204MB   4.64MB      0%   1.44MB   1.45MB
cluster1: cluster.cluster: 5/11/2021 14:18:26
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
Minimums:
    203MB   5.16MB    203MB   4.62MB      0%    555KB    538KB
Averages for 7 samples:
    226MB   6.28MB    225MB   5.22MB      0%   1.08MB   1.07MB
Maximums:
    318MB   8.91MB    317MB   7.44MB      0%   1.56MB   1.47MB

There is barely any cluster traffic other than the normal cluster operations. The “data” and “total” sent/received is nearly identical.

And the latency was an average of .1 ms lower.

Policy Group            Latency
-------------------- ----------

-total-                323.00us
extreme-fixed          323.00us
-total-                323.00us
extreme-fixed          323.00us
-total-                325.00us
extreme-fixed          325.00us
-total-                336.00us
extreme-fixed          336.00us
-total-                325.00us
extreme-fixed          325.00us
-total-                328.00us
extreme-fixed          328.00us
-total-                334.00us
extreme-fixed          334.00us
-total-                341.00us
extreme-fixed          341.00us
-total-                336.00us
extreme-fixed          336.00us
-total-                330.00us
extreme-fixed          330.00us

Try it out and see for yourself! If you have questions or comments, enter them below.

Running PowerShell from Linux to Query SMB Shares in NetApp ONTAP

I recently got a question about how to perform the following scenario:

  • Run a script from Linux that calls PowerShell on a remote Windows client using Kerberos
  • Remote Windows client uses PowerShell to authenticate against an ONTAP SMB share

That’s some Inception-style IT work.

Inception Ending Explained: Christopher Nolan's Endless Spinning | Observer

The issue they were having was that the credentials used to connect to the Windows client weren’t passing through to the ONTAP system. As a result, they’d get “Access Denied” in their script when attempting to access the share. I figured out how to get this working and rather than let that knowledge rot in the far reaches of my brain, I’m writing this up, since in my Google hunt, I found lots of people had similar issues with Linux PowerShell (not necessarily to ONTAP).

This is a known issue with some workarounds listed here:

Making the second hop in PowerShell Remoting

One workaround is to use “Resource-based Kerberos constrained delegation,” where you basically tell the 3rd server to accept delegated credentials from the 2nd server via the PrincipalsAllowedToDelegateToAccount parameter in the ADComputer cmdlets. We’ll cover that in a bit, but first…

WAIT. I can run PowerShell on Linux???

Well, yes! And this article tells you how to install it:

Installing PowerShell on Linux

Now, the downside is that not all PowerShell modules are available from Linux (for example, ActiveDirectory isn’t currently available). But it works!

PS /> New-PSSession -ComputerName COMPUTER -Credential administrator@NTAP.LOCAL -Authentication Kerberos

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **

Id Name     Transport ComputerName ComputerType  State  ConfigurationName    Availability
-- ----     --------- ------------ ------------  ----- --------------------- ------------
9 Runspace9 WSMan     COMPUTER     RemoteMachine Opened Microsoft.PowerShell Available

In that document, they don’t list CentOS/RHEL 8, which can be problematic, as you might run into some issues with the SSL libraries (This blog calls one of those issues out, as well as a few others).

On my Centos8.3 box, I ran into this issue:

New-PSSession: This parameter set requires WSMan, and no supported WSMan client library was found. WSMan is either not installed or unavailable for this system.

Using the guidance from the blog listed earlier, I found that there were a couple of files not found:

# ldd /opt/microsoft/powershell/7/libmi.so
…
libssl.so.1.0.0 => not found
libcrypto.so.1.0.0 => not found
…

That blog lists 1.0.2 as what is needed and looks to be using a different Linux flavor. You can find the files you need/where they live with:

# find / -name 'libssl.so.1.'
/usr/lib64/.libssl.so.1.1.hmac
/usr/lib64/libssl.so.1.1
/usr/lib64/libssl.so.1.1.1g
/usr/lib64/.libssl.so.1.1.1g.hmac
/opt/microsoft/powershell/7/libssl.so.1.0.0

Then you can use the symlink workaround and those files show up properly with ldd:

ln -s libssl.so.1.1 libssl.so.1.0.0
ln -s libcrypto.so.1.1 libcrypto.so.1.0.0
ldd /opt/microsoft/powershell/7/libmi.so
...
libssl.so.1.0.0 => /lib64/libssl.so.1.0.0 (0x00007f41ce3fc000)
libcrypto.so.1.0.0 => /lib64/libcrypto.so.1.0.0 (0x00007f41cdf16000)
...

However, authenticating with the server also requires an additional step.

Authenticating Linux PowerShell with Windows

You can authenticate to Windows servers with Linux PowerShell using the following methods:

Basic Default Kerberos Credssp Digest Negotiate

Here’s how each auth method works (or doesn’t) without doing anything else.

BasicNew-PSSession: Basic authentication is not supported over HTTP on Unix.
DefaultNew-PSSession: MI_RESULT_ACCESS_DENIED
CredSSPNew-PSSession: MI_RESULT_ACCESS_DENIED
DigestNew-PSSession: MI_RESULT_ACCESS_DENIED
KerberosAuthorization failed Unspecified GSS failure.
NegotiateAuthorization failed Unspecified GSS failure.
Auth methods with Linux PowerShell and results with no additional configs

In several places, I’ve seen the recommendation to install gssntlmssp on the Linux client, which works fine for “Negotiate” methods:

PS /> New-PSSession -ComputerName SERVER -Credential administrator@NTAP.LOCAL -Authentication Negotiate

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **

Id Name Transport ComputerName ComputerType State ConfigurationName Availability
-- ---- --------- ------------ ------------ ----- ----------------- ------------
2 Runspace2 WSMan SERVER       RemoteMachine Opened Microsoft.PowerShell Available

But not for Kerberos:

PS /> New-PSSession -ComputerName SERVER -Credential administrator@NTAP.LOCAL -Authentication Kerberos

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **

New-PSSession: [SERVER] Connecting to remote server SERVER failed with the following error message : Authorization failed Unspecified GSS failure. Minor code may provide more information Configuration file does not specify default realm For more information, see the about_Remote_Troubleshooting Help topic.

The simplest way to get around this is to add the Linux client to the Active Directory domain. Then you can use Kerberos for authentication to the client (at least with a user that has the correct permissions, such as a domain administrator).

*Alternately, you could do all this manually, which I don’t recommend.

**For non-AD KDCs, config methods will vary.

# realm join NTAP.LOCAL
Password for Administrator:

# pwsh
PowerShell 7.1.3
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

PS /> New-PSSession -ComputerName SERVER -Credential administrator@NTAP.LOCAL -Authentication Kerberos

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********


 Id Name            Transport ComputerName    ComputerType    State         ConfigurationName     Availability
 -- ----            --------- ------------    ------------    -----         -----------------     ------------
  1 Runspace1       WSMan     SERVER          RemoteMachine   Opened        Microsoft.PowerShell     Available


So, now that we know we can establish a session to the Windows server where we want to leverage PowerShell, now what?

Double-hopping with Kerberos using Delegation

One way to use Kerberos across multiple servers (including NetApp ONTAP) is to leverage the PrincipalsAllowedToDelegateToAccount parameter.

The script I’ll use does a basic “Get-Content” call to a file in an SMB/CIFS share in ONTAP (similar to “cat” in Linux).

If I don’t set the PrincipalsAllowedToDelegateToAccount parameter, a credential passed from Linux PowerShell to a Windows server to ONTAP will use Kerberos -> NTLM (with a NULL user) for the authentication and this is the end result:

# pwsh test.ps1

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********

Test-Path: Access is denied
False
Get-Content: Access is denied
Get-Content: Cannot find path '\\DEMO\files\file-symlink.txt' because it does not exist.

In a packet capture, we can see the session setup uses NULL with NTLMSSP:

14    0.031496   x.x.x.x   x.x.x.y      SMB2 289  Session Setup Request, NTLMSSP_AUTH, User: \

And here’s what the ACCESS_DENIED looks like:

20    0.043026   x.x.x.x   x.x.x.y      SMB2 166  Tree Connect Request Tree: \\DEMO\files
21    0.043217   x.x.x.y   x.x.x.x      SMB2 131  Tree Connect Response, Error: STATUS_ACCESS_DENIED

To use Kerberos passthrough/delegation, I run this PowerShell command to set the parameter on the destination (ONTAP) CIFS server:

Set-ADComputer -Identity DEMO -PrincipalsAllowedToDelegateToAccount SERVER$

That allows the SMB session to ONTAP to set up using Kerberos auth:

2603 26.877660 x.x.x.x x.x.x.y SMB2 2179 Session Setup Request
2673 26.909735 x.x.x.y x.x.x.x SMB2 326 Session Setup Response
supportedMech: 1.2.840.48018.1.2.2 (MS KRB5 - Microsoft Kerberos 5)

And the tree connect succeeds (you may need to run klist purge on the Windows client):

2674 26.910117 x.x.x.x x.x.x.y SMB2 154 Tree Connect Request Tree: \demo\files
2675 26.910630 x.x.x.x x.x.x.y SMB2 138 Tree Connect Response

This is the result from the Linux client:

# pwsh test.ps1

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********

True
This is a file symlink.

So, how do we work around this issue if we can’t delegate Kerberos?

Using the NULL user and NTLM

Remember when I said the request without Kerberos delegation used the NULL user and NTLMSSP?

14    0.031496   x.x.x.x   x.x.x.y      SMB2 289  Session Setup Request, NTLMSSP_AUTH, User: \ 

The reason we saw “Access Denied” to the ONTAP CIFS/SMB share is because ONTAP disallows the NULL user by default. However, in ONTAP 9.0 and later, you can enable NULL user authentication, as described in this KB article:

How to grant access to NULL (Anonymous) user in Clustered Data ONTAP

Basically, it’s a simple two-step process:

  1. Create a name mapping rule for ANONYMOUS LOGON
  2. Set the Windows default NULL user in the CIFS options

Here’s how I did it in my SVM (address is the Windows client IP):

::*> vserver name-mapping create -vserver DEMO -direction win-unix -position 3 -pattern "ANONYMOUS LOGON" -replacement pcuser -address x.x.x.x/24

The Windows user needs to be a valid Windows user.

::*> cifs options modify -vserver DEMO -win-name-for-null-user NTAP\powershell-user

You can verify ONTAP can find it with:

::*> access-check authentication translate -node node1 -vserver DEMO -win-name NTAP\powershell-user
S-1-5-21-3552729481-4032800560-2279794651-1300

Once that’s done, we authenticate with NTLM and get access with the NULL user:

14 0.009012 x.x.x.x x.x.x.y SMB2 289 Session Setup Request, NTLMSSP_AUTH, User: \
27 0.075264 x.x.x.x x.x.x.y SMB2 166 Tree Connect Request Tree: \DEMO\files
28 0.075747 x.x.x.y x.x.x.x SMB2 138 Tree Connect Response

And the Linux client is able to run the PowerShell calls:

# pwsh test.ps1

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********

True
This is a file symlink.

Questions? Comments? Add them below!

MacOS NFS Clients with ONTAP – Tips and Considerations

When I’m testing stuff out for customer deployments that I don’t work with a ton, I like to keep notes on the work so I can reference it later for TRs or other things. A blog is a great place to do that, as it might help other people in similar scenarios. This won’t be an exhaustive list, and it is certain to change over time and possibly make its way into a TR, but here we go…

ONTAP and Multiprotocol NAS

Before I get into the MacOS client stuff, we need to understand ONTAP multiprotocol NAS, as it can impact how MacOS clients behave.

In ONTAP, you can serve the same datasets to clients regardless of the NAS protocol they use (SMB or NFS). Many clients can actually do both protocols – MacOS is one of those clients.

The way ONTAP does Multiprotocol NAS (and keeps permissions predictable) is via name mappings and volume “security styles,” which controls what kind of ACLs are in use. TR-4887 goes into more detail on how all that works, but at a high level:

NTFS security styles use NTFS ACLs

SMB clients will map to UNIX users and NFS clients will require mappings to valid Windows users for authentication. Then, permissions are controlled via ACLs. Chmod/chown from NFS clients will fail.

UNIX security styles use UNIX mode bits (rwx) and/or NFSv4 ACLs

SMB clients will require mappings to a valid UNIX user for permissions; NFS clients will only require mapping to a UNIX user name if using NFSv4 ACLs. SMB clients can do *some* permissions changes, but on a very limited basis.

Mixed security styles always use either UNIX or NTFS effective security styles, based on last ACL change

Basically, if an NFS client chmods a file, it switches to UNIX security style. If an SMB client changes ownership of the file, it flips back to NTFS security style. This allows you to change permissions from any client, but you need to ensure you have proper name mappings in place to avoid undesired permission behavior. Generally, we recommend avoiding mixed security styles in most cases.

MacOS NFS Client Considerations

When using MacOS for an NFS client, there are a few things I’ve run into the past week or two while testing that you would want to know to avoid issues.

MacOS can be configured to use Active Directory LDAP for UNIX Identities

When you’re doing multiprotocol NAS (even if the clients will only do NFS, your volumes might have NTFS style permissions), you want to try to use a centralized name service like LDAP so that ONTAP, SMB clients and NFS clients all agree on who the users are, what groups they belong to, what numeric IDs they have, etc. If ONTAP thinks a user has a numeric ID of 1234 and the client things that user has a numeric ID of 5678, then you likely won’t get the access you expected. I wrote up a blog on configuring MacOS clients to use AD LDAP here:

MacOS clients can also be configured to use single sign on with AD and NFS home directories

Your MacOS clients – once added to AD in the blog post above – can now log in using AD accounts. There’s also an additional tab in the Directory Utility that allows you to auto-create home directories when a new user logs in to the MacOS client.

But you can also configure the auto-created home directories to leverage an NFS mount on the ONTAP storage system. You can configure the MacOS client to automount homedirs and then configure the MacOS client to use that path. (This process varies based on Mac version; I’m on 10.14.4 Catalina)

By default, the homedir path is /home in auto_master. We can use that.

Then, chmod the /etc/auto_home file to 644:

$ sudo chmod 644 /etc/auto_home

Create a volume on the ONTAP cluster for the homedirs and ensure it’s able to be mounted from the MacOS clients via the export policy rules (TR-4067 covers export policy rules):

::*> vol show -vserver DEMO -volume machomedirs -fields junction-path,policy
vserver volume      policy  junction-path
------- ----------- ------- ----------------
DEMO    machomedirs default /machomedirs

Create qtrees for each user and set the user/group and desired UNIX permissions:

qtree create -vserver DEMO -volume machomedirs -qtree prof1 -user prof1 -group ProfGroup -unix-permissions 755
qtree create -vserver DEMO -volume machomedirs -qtree student1 -user student1 -group group1 -unix-permissions 755

(For best results on Mac clients, use UNIX security styles.)

Then modify the automount /etc/auto_home file to use that path for homedir mounts. When a user logs in, the homedir will auto mount.

This is the line I used:

* -fstype=nfs nfs://demo:/machomedirs/&

And I also add the home mount

Then apply the automount change:

$ sudo automount -cv
automount: /net updated
automount: /home updated
automount: /Network/Servers updated
automount: no unmounts

Now, when I cd to /home/username, it automounts that path:

$ cd /home/prof1
$ mount
demo:/machomedirs/prof1 on /home/prof1 (nfs, nodev, nosuid, automounted, nobrowse)

But if I want that path to be the new homedir path, I would need to log in as that user and then go to “System Preferences -> Users and Groups” and right click the user. Then select “Advanced Options.”

Then you’d need to restart. Once that happens, log in again and when you first open Terminal, it will use the NFS homedir path.

NOTE: You may want to test if the Mac client can manually mount the homedir before testing logins. If the client can’t automount the homedir on login things will break.

Alternately, you can create a user with the same name as the AD account and then modify the homedir path (this removes the need to login). The Mac will pick up the correct UID, but the group ID may need to be changed.

If you use SMB shares for your home directories, it’s as easy as selecting “Use UNC path” in the User Experience area of Directory Utility (there’s no way to specify NFS here):

With new logins, the profile will get created in the qtree you created for the homedir (and you’ll go through the typical initial Mac setup screens):

# ls -la
total 28
drwxrwxr-x 6 student1 group1 4096 Apr 14 16:39 .
drwxr-xr-x 6 root root 4096 Apr 14 15:28 ..
drwx------ 2 student1 group1 4096 Apr 14 16:39 Desktop
drwx------ 2 student1 group1 4096 Apr 14 16:35 Downloads
drwxr-xr-x 25 student1 group1 4096 Apr 14 16:39 Library
-rw-r--r-- 1 student1 group1 4096 Apr 14 16:35 ._Library
drwx------ 4 student1 group1 4096 Apr 14 16:35 .Spotlight-V100

When you open terminal, it automounts the NFS home directory mount for that user and drops you right into your folder!

Mac NFS Considerations, Caveats, Issues

If you’re using NFS on Mac clients, there are two main things to remember:

  • Volumes/qtrees using UNIX security styles work best with NFS in general
  • Terminal/CLI works better than Finder in nearly all instances

If you have to/want to use Finder, or you have to/want to use NTFS security styles for multiprotocol, then there are some things you’d want to keep in mind.

  • If possible, connect the Mac client to the Active Directory domain and use LDAP for UNIX identities as described above.
  • Ensure your users/groups are all resolving properly on the Mac clients and ONTAP system. TR-4887 and TR-4835 cover some commands you can use to check users and groups, name mappings, group memberships, etc.
  • If you’re using NTFS security style volumes/qtrees and want the Finder to work properly for copies to and from the NFS mount, configure the NFS export policy rule to set -ntfs-unix-security-ops to “ignore” – Finder will bail out if ONTAP returns an error, so we want to silently fail those operations (such as SETATTR; see below).
  • When you open a file for reading/writing (such as a text file), Mac creates a ._filename file along with it. Depending on how many files you have in your volume, this can be an issue. For example, if you open 1 million files and Mac creates 1 million corresponding ._filename files, that starts to add up. Don’t worry! You’re not alone: https://apple.stackexchange.com/questions/14980/why-are-dot-underscore-files-created-and-how-can-i-avoid-them
  • If you’re using DFS symlinks, check out this KB: DFS links do not work on MAC OS client, with ONTAP 9.5 and symlinks enabled

I’ve also run into some interesting behaviors with Mac/Finder/SMB and junction paths in ONTAP, as covered in this blog:

Workaround for Mac Finder errors when unzipping files in ONTAP

One issue that I did a pretty considerable amount of analysis on was the aforementioned “can’t copy using Finder.” Here are the dirty details…

Permissions Error When Copying a File to a NFS Mount in ONTAP using Finder

In this case, a file copy worked using Terminal, but was failing with permissions errors when using Finder and complaining about the file already existing.

First, it wants a login (which shouldn’t be needed):

Then it says this:

If you select “Replace” this is the error:

If you select “Stop” it stops and you are left with an empty 0 byte “file” – so the copy failed.

If you select “Keep Both” the Finder goes into an infinite loop of 0 byte file creations. I stopped mine at around 2500 files (forced an unmount):

# ls -al | wc -l
1981
# ls -al | wc -l
2004
# ls -al | wc -l
2525

So what does that happen? Well, in a packet trace, I saw the following:

The SETATTR fails on CREATE (expected in NFS operations on NTFS security style volumes in ONTAP, but not expected for NFS clients as per RFC standards):

181  60.900209  x.x.x.x    x.x.x.y      NFS  226  V3 LOOKUP Call (Reply In 182), DH: 0x8ec2d57b/copy-file-finder << Mac NFS client checks if the file exists
182  60.900558  x.x.x.y   x.x.x.x      NFS  186  V3 LOOKUP Reply (Call In 181) Error: NFS3ERR_NOENT << does not exist, so let’s create it!
183  60.900633  x.x.x.x    x.x.x.y      NFS  238  V3 CREATE Call (Reply In 184), DH: 0x8ec2d57b/copy-file-finder Mode: EXCLUSIVE << creates the file
184  60.901179  x.x.x.y   x.x.x.x      NFS  362  V3 CREATE Reply (Call In 183)
185  60.901224  x.x.x.x    x.x.x.y      NFS  238  V3 SETATTR Call (Reply In 186), FH: 0x7b82dffd
186  60.901564  x.x.x.y   x.x.x.x      NFS  214  V3 SETATTR Reply (Call In 185) Error: NFS3ERR_PERM << fails setting attributes, which also fails the copy of the actual file data, so we have a 0 byte file

Then it REMOVES the file (since the initial operation fails) and creates it again, and SETATTR fails again. This is where that “Keep Both” loop behavior takes place.

229 66.995698 x.x.x.x x.x.x.y NFS 210 V3 REMOVE Call (Reply In 230), DH: 0x8ec2d57b/copy-file-finder
233 67.006816 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 234), DH: 0x8ec2d57b/copy-file-finder
234 67.007166 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 233) Error: NFS3ERR_NOENT
247 67.036056 x.x.x.x x.x.x.y NFS 238 V3 CREATE Call (Reply In 248), DH: 0x8ec2d57b/copy-file-finder Mode: EXCLUSIVE
248 67.037662 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 247)
249 67.037732 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 250), FH: 0xc33bff48
250 67.038534 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 249) Error: NFS3ERR_PERM

With Terminal, it operates a little differently. Rather than bailing out after the SETATTR failure, it just retries it:

11 19.954145 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 12), DH: 0x8ec2d57b/copy-file-finder
12 19.954496 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 11) Error: NFS3ERR_NOENT
13 19.954560 x.x.x.x x.x.x.y NFS 226 V3 LOOKUP Call (Reply In 14), DH: 0x8ec2d57b/copy-file-finder
14 19.954870 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 13) Error: NFS3ERR_NOENT
15 19.954930 x.x.x.x x.x.x.y NFS 258 V3 CREATE Call (Reply In 18), DH: 0x8ec2d57b/copy-file-finder Mode: UNCHECKED
16 19.954931 x.x.x.x x.x.x.y NFS 230 V3 LOOKUP Call (Reply In 17), DH: 0x8ec2d57b/._copy-file-finder
17 19.955497 x.x.x.y x.x.x.x NFS 186 V3 LOOKUP Reply (Call In 16) Error: NFS3ERR_NOENT
18 19.957114 x.x.x.y x.x.x.x NFS 362 V3 CREATE Reply (Call In 15)
25 19.959031 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 26), FH: 0x8bcb16f1
26 19.959512 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 25) Error: NFS3ERR_PERM
27 19.959796 x.x.x.x x.x.x.y NFS 238 V3 SETATTR Call (Reply In 28), FH: 0x8bcb16f1 << Hey let's try again and ask in a different way!
28 19.960321 x.x.x.y x.x.x.x NFS 214 V3 SETATTR Reply (Call In 27)

The first SETATTR tries to chmod to 700:

Mode: 0700, S_IRUSR, S_IWUSR, S_IXUSR

The retry uses 777. Since the file already shows as 777, it succeeds (because it was basically fooled):

Mode: 0777, S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTH

Since Finder bails on the error, setting the NFS server to return no error here for this export (ntfs-unix-security-ops ignore) on this client allows the copy to succeed. You can create granular rules in your export policy rules to just set that option for your Mac clients.

Now, why do our files all show as 777?

Displaying NTFS Permissions via NFS

Because NFS doesn’t understand NTFS permissions, the job to translate user identities into valid access rights falls onto the shoulders of ONTAP. A UNIX user maps to a Windows user and then that Windows user is evaluated against the folder/file ACLs.

So “777” here doesn’t mean we have wide open access; we only have access based on the Windows ACL. Instead, it just means “the Linux client can’t view the access level for that user.” In most cases, this isn’t a huge problem. But sometimes, you need files/folders not to show 777 (like for applications that don’t allow 777).

In that case, you can control somewhat how NFS clients display NTFS ACLs in “ls” commands with the NFS server option ntacl-display-permissive-perms.

[-ntacl-display-permissive-perms {enabled|disabled}] - Display maximum NT ACL Permissions to NFS Client (privilege: advanced)
This optional parameter controls the permissions that are displayed to NFSv3 and NFSv4 clients on a file or directory that has an NT ACL set. When true, the displayed permissions are based on the maximum access granted by the NT ACL to any user. When false, the displayed permissions are based on the minimum access granted by the NT ACL to any user. The default setting is false.

The default setting of “false” is actually “disabled.” When that option is enabled, this is the file/folder view:

When that option is disabled (the default):

This option is covered in more detail in TR-4067, but it doesn’t require a remount to take effect. It may take some time for the access caches to clear to see the results, however.

Keep in mind that these listings are approximations of the access as seen by the current user. If the option is disabled, you see the minimum access; if the option is enabled, you see the maximum access. For example, the “test” folder above shows 555 when the option is disabled, but 777 when the option is enabled.

These are the actual permissions on that folder:

::*> vserver security file-directory show -vserver DEMO -path /FG2/test
Vserver: DEMO
File Path: /FG2/test
File Inode Number: 10755
Security Style: ntfs
Effective Style: ntfs
DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
UNIX User Id: 1102
UNIX Group Id: 10002
UNIX Mode Bits: 777
UNIX Mode Bits in Text: rwxrwxrwx
ACLs: NTFS Security Descriptor
Control:0x8504
Owner:BUILTIN\Administrators
Group:NTAP\ProfGroup
DACL - ACEs
ALLOW-Everyone-0x1200a9-OI|CI (Inherited)
ALLOW-NTAP\prof1-0x1f01ff-OI|CI (Inherited)

Here are the expanded ACLs:

                     Owner:BUILTIN\Administrators
                     Group:NTAP\ProfGroup
                     DACL - ACEs
                       ALLOW-Everyone-0x1200a9-OI|CI (Inherited)
                          0... .... .... .... .... .... .... .... = Generic Read
                          .0.. .... .... .... .... .... .... .... = Generic Write
                          ..0. .... .... .... .... .... .... .... = Generic Execute
                          ...0 .... .... .... .... .... .... .... = Generic All
                          .... ...0 .... .... .... .... .... .... = System Security
                          .... .... ...1 .... .... .... .... .... = Synchronize
                          .... .... .... 0... .... .... .... .... = Write Owner
                          .... .... .... .0.. .... .... .... .... = Write DAC
                          .... .... .... ..1. .... .... .... .... = Read Control
                          .... .... .... ...0 .... .... .... .... = Delete
                          .... .... .... .... .... ...0 .... .... = Write Attributes
                          .... .... .... .... .... .... 1... .... = Read Attributes
                          .... .... .... .... .... .... .0.. .... = Delete Child
                          .... .... .... .... .... .... ..1. .... = Execute
                          .... .... .... .... .... .... ...0 .... = Write EA
                          .... .... .... .... .... .... .... 1... = Read EA
                          .... .... .... .... .... .... .... .0.. = Append
                          .... .... .... .... .... .... .... ..0. = Write
                          .... .... .... .... .... .... .... ...1 = Read

                       ALLOW-NTAP\prof1-0x1f01ff-OI|CI (Inherited)
                          0... .... .... .... .... .... .... .... = Generic Read
                          .0.. .... .... .... .... .... .... .... = Generic Write
                          ..0. .... .... .... .... .... .... .... = Generic Execute
                          ...0 .... .... .... .... .... .... .... = Generic All
                          .... ...0 .... .... .... .... .... .... = System Security
                          .... .... ...1 .... .... .... .... .... = Synchronize
                          .... .... .... 1... .... .... .... .... = Write Owner
                          .... .... .... .1.. .... .... .... .... = Write DAC
                          .... .... .... ..1. .... .... .... .... = Read Control
                          .... .... .... ...1 .... .... .... .... = Delete
                          .... .... .... .... .... ...1 .... .... = Write Attributes
                          .... .... .... .... .... .... 1... .... = Read Attributes
                          .... .... .... .... .... .... .1.. .... = Delete Child
                          .... .... .... .... .... .... ..1. .... = Execute
                          .... .... .... .... .... .... ...1 .... = Write EA
                          .... .... .... .... .... .... .... 1... = Read EA
                          .... .... .... .... .... .... .... .1.. = Append
                          .... .... .... .... .... .... .... ..1. = Write
                          .... .... .... .... .... .... .... ...1 = Read

So, prof1 has Full Control (7) and “Everyone” has Read (5). That’s where the minimum/maximum permissions show up. So you won’t get *exact* permissions here. If you want exact permission views, consider using UNIX security styles.

DS_Store files

Mac will leave these little files laying around as users browse shares. In a large environment, that can start to create clutter, so you may want to consider disabling the creation of these on network shares (such as NFS mounts), as per this:

http://hints.macworld.com/article.php?story=2005070300463515http://hints.macworld.com/article.php?story=2005070300463515

If you have questions, comments or know of some other weirdness in MacOS with NFS, comment below!

How to Configure MacOS to Use Active Directory LDAP for UNIX users/groups

In NetApp ONTAP, it’s possible to serve data to NAS clients over SMB and NFS, including the same datasets. This is known as “multiprotocol NAS” and I cover the best practices for that in the new TR-4887:

TR-4887: Multiprotocol NAS Best Practices in ONTAP

When you do multiprotocol NAS in ONTAP (or really, and storage system), it’s usually best to leverage a centralized repository for user names, group names and numeric IDs that the NAS clients and NAS servers all point to (such as LDAP). That way, you get no surprises when accessing files and folders – user and groups get the expected ownership and permissions.

I cover LDAP in ONTAP in TR-4835:

TR-4835: LDAP in NetApp ONTAP

One of the more commonly implemented solutions for LDAP in environments that serve NFS and SMB is Active Directory. In these environments, you can either use either native UNIX attributes or a 3rd party utility, such as Centrify. (Centrify basically just uses the AD UNIX attributes and centralizes the management into a GUI – both are covered in the LDAP TR.)

While most Linux clients are fairly straightforward for LDAP integration, MacOS does things slightly differently. However, it’s pretty easy to configure.

Note: The steps may vary based on your environment configs and this covers just AD LDAP; not OpenLDAP/IDM or other Linux-based LDAP.

macOS Big Sur is here - Apple

Step 1: Ensure the Mac is configured to use the same DNS as the AD domain

This is done via “Network” settings in System Preferences. DNS is important here because it will be used to query the AD domain when we bind the Mac for the Directory Services. In the following, I’ve set the DNS server IP and the search domain to my AD domain “NTAP.LOCAL”:

Then I tested the domain lookup in Terminal:

Step 2: Configure Directory Services to use Active Directory

The “Directory Utility” is what we’ll use here. It’s easiest to use the spotlight search to find it.

Essentially, this process adds the MacOS to the Active Directory domain (as you would with a Windows server or a Linux box with “realm join”) and then configures the LDAP client on the Mac to leverage specific attributes for LDAP queries.

In the above, I’ve used uidNumber and gidNumber as the attributes. You can also control/view these via the CLI command “dsconfigad”:

Configure domain access in Directory Utility on Mac

I can see in my Windows AD domain the machine account was created:

A few caveats here about the default behavior for this:

  • LDAP queries will be encrypted by default, so if you’re trying to troubleshoot via packet capture, you won’t see a ton of useful info (such as attributes used for queries). To disable this (mainly for troubleshooting purposes):
$ dsconfigad -packetsign disable -packetencrypt disable
  • MacOS uses sAMAccountName as the user name/uid value, so it should work fine with AD out of the gate
  • MacOS adds additional Mac-specific system groups to the “id” output (such as netaccounts:62, and GIDs 701/702); these may need to be added to LDAP, depending on file ownership
  • LDAP queries to AD from Mac will use the Global Catalog port 3268 by default when using Active Directory config (which I was able to see from a packet capture)

This use of the global catalog port can be problematic, as standard LDAP configurations in AD don’t set the Global Catalog to replicate attributes, but rather uses the standard port 389/636 for LDAP communication. AD doesn’t replicate the UNIX attributes across the global catalog by default for LDAP, so you’d have to configure that manually (covered in TR-4835) or modify the port the Mac uses for LDAP.

My AD domain does have the attributes that replicate via the Global Catalog, so the LDAP lookups work for me from the Mac:

Here’s what the prof1 user looks like from a CentOS client:

# id prof1
uid=1102(prof1) gid=10002(ProfGroup) groups=10002(ProfGroup),10000(Domain Users),48(apache-group),1101(group1),1202(group2),1203(group3)

This is how that user looks from the ONTAP CLI:

cluster::> set advanced; access-check authentication show-creds -node node1 -vserver DEMO -unix-user-name prof1 -list-name true -list-id true
UNIX UID: 1102 (prof1) <> Windows User: S-1-5-21-3552729481-4032800560-2279794651-1110 (NTAP\prof1 (Windows Domain User))
GID: 10002 (ProfGroup)
Supplementary GIDs:
10002 (ProfGroup)
10000 (Domain Users)
1101 (group1)
1202 (group2)
1203 (group3)
48 (apache-group)
Primary Group SID: S-1-5-21-3552729481-4032800560-2279794651-1111 NTAP\ProfGroup (Windows Domain group)
Windows Membership:
S-1-5-21-3552729481-4032800560-2279794651-1301 NTAP\apache-group (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1106 NTAP\group2 (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-513 NTAP\DomainUsers (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1105 NTAP\group1 (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1107 NTAP\group3 (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1111 NTAP\ProfGroup (Windows Domain group)
S-1-5-21-3552729481-4032800560-2279794651-1231 NTAP\local-group.ntap (Windows Alias)
S-1-18-2 Service asserted identity (Windows Well known group)
S-1-5-32-551 BUILTIN\Backup Operators (Windows Alias)
S-1-5-32-544 BUILTIN\Administrators (Windows Alias)
S-1-5-32-545 BUILTIN\Users (Windows Alias)
User is also a member of Everyone, Authenticated Users, and Network Users
Privileges (0x22b7):
SeBackupPrivilege
SeRestorePrivilege
SeTakeOwnershipPrivilege
SeSecurityPrivilege
SeChangeNotifyPrivilege

Most people aren’t going to want/be allowed to crack open ADSIEdit and modify schema attributes, so you’d want to change how MacOS queries LDAP to use port 389 or 636. I’m currently waiting on word of how to do that from Apple, so I’ll update when I get that info. If you are reading this and already know, feel free to add to the comments!

Step 3: Mount the NetApp NFS export and test multiprotocol access

NFS mounts to UNIX security style volumes are pretty straightforward, so we won’t cover that here. Where it gets tricky is when your ONTAP volumes are NTFS security style. When that’s the case, a UNIX -> Windows name mapping occurs when using NFS, as we need to make sure the user trying to access the NTFS permissions truly has access.

This is the basic process:

  • MacOS NFS client sends a numeric UID and GID to the NetApp ONTAP system (if NFSv3 is used)
  • If the volume is NTFS security, ONTAP will try to translate the numeric IDs into user names. The method depends on the cluster config; in this case, we’ll use LDAP.
  • If the numeric IDs map to user names/group names, then ONTAP uses those UNIX names and tries to find a valid Windows name with the same names; if none exist, ONTAP looks for explicit name mapping rules and a default Windows user; if none of those work then access is denied.

I have mounted a volume to my MacOS client that uses NTFS security style.

This is the volume in ONTAP:

::*> vol show -vserver DEMO -volume FG2 -fields security-style
vserver volume security-style
------- ------ --------------
DEMO       FG2           ntfs

MacOS user IDs start at the ID 501; So my “admin” user ID is 501. This user doesn’t exist in LDAP.

ONTAP has a local user named “Podcast” but no valid Windows user mapping:

::*> set advanced; access-check authentication show-creds -node ontap9-tme-8040-01 -vserver DEMO -uid 501
(vserver services access-check authentication show-creds)
Vserver: DEMO (internal ID: 10)
Error: Get user credentials procedure failed
[ 33] Determined UNIX id 501 is UNIX user 'Podcast'
[ 34] Using a cached connection to ntap.local
[ 36] Trying to map 'Podcast' to Windows user 'Podcast' using
implicit mapping
[ 36] Successfully connected to ip 10.193.67.236, port 445
using TCP
[ 46] Successfully authenticated with DC oneway.ntap.local
[ 49] Could not find Windows name 'Podcast'
[ 49] Unable to map 'Podcast'. No default Windows user defined.
**[ 49] FAILURE: Name mapping for UNIX user 'Podcast' failed. No
** mapping found
Error: command failed: Failed to get user credentials. Reason: "SecD Error: Name mapping does not exist".

In addition, MacOS disables root by default:

https://support.apple.com/en-us/HT204012

So when I try to access this mount, it will attempt to use UID 501 and translate it to a UNIX user and then to a Windows user. Since ONTAP can’t translate UID 501 to a valid Windows user, this will fail and we’ll see it in the event log of the ONTAP CLI.

Here’s the access failure:

Here’s the ONTAP error:

ERROR secd.nfsAuth.noNameMap: vserver (DEMO) Cannot map UNIX name to CIFS name. Error: Get user credentials procedure failed
[ 33] Determined UNIX id 501 is UNIX user 'Podcast'
[ 34] Using a cached connection to ntap.local
[ 36] Trying to map 'Podcast' to Windows user 'Podcast' using implicit mapping
[ 36] Successfully connected to ip 10.193.67.236, port 445 using TCP
[ 46] Successfully authenticated with DC oneway.ntap.local
[ 49] Could not find Windows name 'Podcast'
[ 49] Unable to map 'Podcast'. No default Windows user defined.
**[ 49] FAILURE: Name mapping for UNIX user 'Podcast' failed. No mapping found

When I “su” to a user that *does* have a valid Windows user (such as prof1), this works fine and I can touch a file and get the proper owner/group:

Note that in the above, we see “root:wheel” owned folders; just because root is disabled by default on MacOS doesn’t mean that MacOS isn’t aware of the user. Those folders were created on a separate NFS client.

Also, note in the above that the file shows 777 permissions; this is because those are the allowed permissions for the prof1 user on that file. The permissions are defined by Windows/NTFS. Here, they are set to “Everyone:Full Control” by way of file inheritance. These are the new permissions. Profgroup (with prof1 and studen1 as members) gets write access. Administrator gets “Full Control.” Group10 (with only student1 as a member) gets read access.

In ONTAP, you can also control the way NTFS security style files are viewed on NFS clients with the NFS server option -ntacl-display-permissive-perms. TR-4887 covers that option in more detail.

Prof1 access view after permissions change (write access):

Student1 access view after permissions change (read access only via group ACL):

Read works, but write does not (by design!)

Student2 access view (write access defined by membership in ProfGroup):

Newuser1 access view (not a member of any groups in the ACL):

Newuser1 can create a new file, however, and it shows the proper owner. The permissions are 777 because of the inherited NTFS ACLs from the share:

As you can see, we will get the expected access for users and groups on Mac NFS using NTFS security styles, but the expected *views* won’t always line up. This is because there isn’t a direct correlation between NTFS and UNIX permissions, so we deliver an approximation. ONTAP doesn’t store ACLs for both NTFS and UNIX on disk; it only chooses one or the other. If you require exact NFS permission translation via Mac NFS, consider using UNIX security style and mode bits.

Addendum: Squashing root

In the event your MacOS users enable the root account and become the “root” user on the client, you can squash the root user to an anonymous user by using ONTAP’s export policies and rules. TR-4067 covers how to do this:

NFS Best Practices in ONTAP

Let me know if you have questions!

Brand new tech report: Multiprotocol NAS Best Practices in ONTAP

I don’t like to admit to being a procrastinator, but…

Lazy Sloth Drawing (Page 1) - Line.17QQ.com
(Not actually a sloth)

Four years ago, I said this:

And people have asked about it a few times since then. To be fair, I did say “will be a ways out…”

In actuality, I started that TR in March of 2017. And then again in February of 2019. And then started all over when the pandemic hit, because what else did I have going on? 🙂

And it’s not like I haven’t done *stuff* in that time.

The trouble was, I do multiprotocol NAS every day, so I think I had writer’s block because I didn’t know where to start and the challenge of writing an entire TR on the subject without making it 100-200 pages like some of the others I’ve written was… daunting. But, it’s finally done. And the actual content is under 100 pages!

Topics include:

  • NFS and SMB best practices/tips
  • Name mapping explanations and best practices
  • Name service information
  • CIFS Symlink information
  • Advanced multiprotocol NAS concepts

Multiprotocol NAS Best Practices in ONTAP

If you have any comments/questions, feel free to comment!

Behind the Scenes – Episode 270: NetApp ONTAP File Systems Analytics

Welcome to the Episode 270, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, we do a deep dive into the new ONTAP 9.8 feature in System Manager – File Systems Analytics – and why it’s improving the lives of storage administrators.

Podcast Transcriptions

If you want an AI transcribed copy of the episode, check it out here (just set expectations accordingly):

Episode 270: NetApp ONTAP File Systems Analytics – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes – Episode 269: Medical Imaging Workloads with NetApp Solutions

Welcome to the Episode 269, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, NetApp’s Tony Turner (@tnyturner) drops by to talk about how NetApp solutions are driving successful deployments for medical imaging workloads.

For more information:

FlexPod for Medical Imaging
https://www.netapp.com/pdf.html?item=/media/19793-tr-4865.pdf

Podcast Transcriptions

If you want an AI transcribed copy of the episode, check it out here (just set expectations accordingly):

Episode 269: Medical Imaging Workloads with NetApp Solutions – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes – Episode 261: How DreamWorks Animation Creates New Worlds on NetApp ONTAP

Welcome to the Episode 261, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, I was lucky enough to snag a long-time NetApp customer and engineering partner DreamWorks Animation to talk about how the studio leverages NetApp ONTAP to build some of the most creative and effects intensive CG animated films on the planet.

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 261: How DreamWorks Animation Creates New Worlds on NetApp ONTAP – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

New XCP 1.6.1 Version is now Available!

flash

If you use XCP (or have used it), you might have run into an issue or two on occasion – generally related to memory issues or problems with XCP sync taking too long.

The new XCP 1.6.1 release aims to fix some of those issues.

Key enhancements

Sync Performance : Identified sync bottlenecks w.r.t change handling and fixed in 1.6.1 to Improve the sync performance. Now incremental sync takes less time than baseline.  This would help performing more frequent syncs or will need less sync/cutover window.

Memory Optimization : Memory allocation was a challenge in the previous versions and used to receive “Memory allocation” errors during the large sync jobs. XCP 1.6.1 handles the memory allocation elegantly to further assist in accomplishing the sync job.

Syslog integration : Configure Syslog option to send XCP events to the remote syslog server for monitoring the XCP jobs.

Complete Features List of NetApp XCP 1.6.1

  • Sync performance improvements (NFS/SMB)
  • Memory optimization (NFS Sync)
  • File Analytics (General Availability)
  • Event logging – XCP now supports logging events to xcp_event.log file
  • Syslog support (NFS/SMB)

The XCP TME Karthik Nagalingam also wrote up a blog on XCP:

https://blog.netapp.com/data-migration-xcp

File Analytics

In XCP 1.6, a new File Analytics Engine was added to XCP. This basically configures an Apache web server, a postgresql database and adds a web GUI to view file information using an XCP listener agent. Karthik also wrote a blog on that:

https://blog.netapp.com/xcp-data-migration-software

With this feature, you can add your NFS server/volume and XCP will run periodic scans on the datasets to provide information.

xcp_fsa_add_nfs

Once you add a server, you get a list of the available exports:

xcp_fsa_shares

Once you have the shares, you can click on one and run a scan:

xcp_fsa_home

Once that’s done, you can access the Stat View and File Distribution tabs:

The stat view includes:

  • Hot/cold data size and count
  • File capacity sorted by size
  • File distribution by size
  • Directory depth
  • Directory entries
  • Accessed time
  • Information on space occupied by users

Postgresql DB

In addition to the web GUI, you also can access the postgresql database. That way you can query the database if you choose.

When XCP configures the database, a new table called xcp_cmdb is added:

# sudo -u postgres psql
psql (9.2.24)
Type "help" for help.

postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
xcp_cmdb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(4 rows)

You can access that database and query which tables are added:

# sudo -u postgres psql xcp_cmdb
psql (9.2.24)
Type "help" for help.

xcp_cmdb=# \dt
List of relations
Schema | Name | Type | Owner
--------+---------------------+-------+---------
public | xcp_agent | table | central
public | xcp_login | table | central
public | xcp_nfs_file_server | table | central
public | xcp_nfs_jobs | table | central
public | xcp_nfs_stats | table | central
public | xcp_nfs_tree | table | central
public | xcp_smb_file_server | table | central
public | xcp_smb_jobs | table | central
public | xcp_smb_stats | table | central
public | xcp_smb_tree | table | central
(10 rows)

You can use standard Postgresql commands to query the tables.

xcp_cmdb=# SELECT file_server_ip FROM xcp_nfs_file_server;
file_server_ip
----------------
10.10.10.10
(1 row)


xcp_cmdb=# SELECT * FROM xcp_nfs_jobs;
id | export_path | agent_id | process_group_id | start_timestamp |
end_timestamp | type | status | statistics |
stdout | stderr | log | protocol | files_to_process
--------------------------------------+-------------------------+--------------------------------------+------------------+----------------------------+-----
-----------------------+------+----------+------------------------------------------------------------------------------------------------------------------+
--------+--------+-----+----------+------------------
745a0615-4177-4d83-b945-8e302efc89e9 | 10.10.10.10:techontap | bf83aced-81cd-4b7f-be5d-94e55df11484 | 6562 | 2020-07-27 06:26:55.980298 | 2020
-07-27 06:26:56.785463 | SCAN | FINISHED | {"received": 1699564, "scanned": 6928, "sent_throughput": 138247, "sent": 106400, "receive_throughput": 2208273} |
stdout | stderr | log | NFS | 9047
9a5fc78e-7260-47dd-8cb5-da1820dbcd3e | 10.10.10.10:/home | bf83aced-81cd-4b7f-be5d-94e55df11484 | 2039 | 2020-07-31 00:34:50.001866 | 2020
-07-31 00:34:50.390331 | SCAN | FINISHED | {"received": 9808, "scanned": 21, "sent_throughput": 5627, "sent": 2072, "receive_throughput": 26638} |
stdout | stderr | log | NFS | 839
2767d48c-be49-4232-9a49-9632878bd4c6 | 10.10.10.10:techontap | bf83aced-81cd-4b7f-be5d-94e55df11484 | 5832 | 2020-07-27 06:17:47.496321 | 2020
-07-27 06:17:48.290484 | SCAN | FINISHED | {"received": 1699564, "scanned": 6928, "sent_throughput": 83533, "sent": 106400, "receive_throughput": 1334304} |
stdout | stderr | log | NFS | 9114
(3 rows)

Installation/Upgrade Note

If you’re already running an XCP version prior to 1.6.1, you’ll need to re-run the configure.sh script. This creates the XCP service module on the client (NFS version) to provide better overall management.

For example, if I simply untar the XCP install (like you would do in prior versions to upgrade), then XCP commands work fine, but the new File Systems Analytics module doesn’t, as it needs the XCP service to run properly.

# tar -xvf NETAPP_XCP_1.6.1.tgz
./
./xcp/
./xcp/linux/
./xcp/linux/xcp
./xcp/windows/
./xcp/windows/xcp.exe
./xcp/xcp_gui/
./xcp/xcp_gui/xcp/
./xcp/configure.sh
./xcp/README

# xcp activate
XCP 1.6.1; (c) 2020 NetApp, Inc.; Licensed to Justin Parisi [NetApp Inc] until Thu Oct 22 12:31:11 2020

XCP already activated

Trying to start the listener will fail:

# ./linux/xcp --listen &
[1] 1525
[root@XCP xcp]# Traceback (most recent call last):
File "../xcp.py", line 957, in <module>
File "../xcp.py", line 902, in main
File "../rest/config.py", line 117, in <module>
File "../rest/config.py", line 112, in singleton
File "../rest/config.py", line 163, in __init__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3215, in first
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3339, in _execute_and_instances
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3354, in _get_bind_args
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3332, in _connection_from_session
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1123, in connection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1129, in _connection_for_bind
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 430, in _connection_for_bind
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 354, in connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
File "/local/build/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
File "/local/build/lib/python2.7/site-packages/pgdb.py", line 1623, in connect
sqlalchemy.exc.InternalError: (pg.InternalError) could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?

(Background on this error at: http://sqlalche.me/e/2j85)

[1]+ Exit 1 ./linux/xcp --listen

When we look for the XCP service, it’s not there. That gets created with the script:

# service xcp status
Redirecting to /bin/systemctl status xcp.service
Unit xcp.service could not be found.
# systemctl status xcp
Unit xcp.service could not be found.
# sudo systemctl start xcp
Failed to start xcp.service: Unit not found.

Here’s an example of the configure.sh script to upgrade:

# ./configure.sh 
------------XCP CONFIGURATION SCRIPT -------------
# Checking if xcp service is running

Menu Help:
# Install/ Upgrade -
# Install - Installs PostgreSQL and httpd if not installed and does setup for XCP File Analytics.
# Upgrade - Upgrades takes a backup of xcp folder under /var/www/html/ to xcp_backup
# Repair - Restarts PostgreSQL database, http and XCP services incase if any of them are stopped.
# Cleanup - Uninstalls PostgreSQL and httpd.

1. Installation/Upgrade
2. Repair
3. Cleanup
0. Exit

Please select the option from the menu [0-3]: 1 
--------Setting up Postgres -----------
# Checking if postgres is installed
Redirecting to /bin/systemctl status postgresql.service
# Postgres service is already running 
--------------Setting up httpd -------------
# Checking if httpd is installed
httpd-2.4.6-93.el7.centos.x86_64
# HTTPD is already installed... 
--------------Setting up https -------------
mod_ssl-2.4.6-93.el7.centos.x86_64
# Https is already installed... 
--------------Configuring xcp --------------
# Copying xcp files.....
# Taking backup of xcp folder /var/www/html/xcp in /var/www/html/xcp_backup
NOTE:
If you are running for first time, select option 1.
If you are upgrading select option 0.
______________________________
Please choose the menu you want to start:
1. Configure client system
2. Reset admin password
3. Reset Postgres database password
4. Rebuild xcp.ini file
5. Delete database and reconfigure client system

0. Quit
-------------------------------
Enter digit between [0-5] >> 1

# Postgres service is running
# Database already exists and configured. 
---------------Starting xcp service -------------
# Creating xcp service...
Redirecting to /bin/systemctl status xcp.service
# XCP service started successfully
Created symlink from /etc/systemd/system/multi-user.target.wants/xcp.service to /etc/systemd/system/xcp.service.
root 1818 1 19 20:03 ? 00:00:00 /usr/bin/xcp --listen
postgres 1833 1587 0 20:03 ? 00:00:00 postgres: central xcp_cmdb 127.0.0.1(34018) idle 
---------------------------------------------
XCP File analytics is configured successfully
To start XCP service manually, use "sudo systemctl start xcp" command

Use following link and login as 'admin' user for GUI access
https://XCP/xcp 
---------------------------------------------

Once that’s done, XCP is ready to use!

Full NetApp Support

Since XCP 1.5, you can call in for official NetApp Support for XCP.

Support Site Self-Service Options

To review, order, and renew software subscriptions, click My Service Contracts & Warranties on the Service and Support page of the NetApp Support site.