Getting ONTAP NFS Kerberos to Work with FreeIPA

Image result for hamburglar

Obviously, with the social distancing/lockdowns happening, I have had more time to write up blogs on things. So, here’s another one. If you have suggestions for topics, let me know and I’ll see about writing them up.

Before we get to the meat of what the title is (or scroll down to the bottom if you want the quick and dirty steps), let’s recap NFS security and Kerberos…

NFS is a protocol that allows multiple clients communicate to a single storage endpoint as a way to share data across a network. Because data is transmitted over a network, it’s important to be able to secure that data, both at-rest and in-flight.

At-rest Security in NFS

At-rest security refers to security applied to data residing on the storage system, as well as the interaction between NFS client and NFS server to negotiate things like NFS version, identity, etc.

At-rest security for NFS includes:

  • Export policies and rules
  • Permissions/ACLs
  • User/group owners
  • Name ID strings (NFSv4.0 and later)

However, when data passes over the wire, packet sniffers are able to see identifying information like IP addresses, users, groups, permissions, file names, paths, etc. All it would take is someone being able to see this information to easily create duplicate set ups to “spoof” users and groups and gain access to data. At-rest security mostly protects you from threats that don’t have this knowledge or expertise. To prevent bad actors from getting information from the transmission of data, you’d need to set up in-flight security.

In-flight Security in NFS

In-flight security refers to securing the actual data packets in transit.

For NFS, this can be done in a few ways:

For ONTAP, tunneling NFS over SSH or stunnel isn’t supported, but you can use NFS Kerberos.

For a deeper look at NFS Kerberos in ONTAP, see:

NFS Kerberos in ONTAP Primer

Kerberos Security Flavors

NFS Kerberos has 3 methods that can be used to secure things. Each provides an additional level of security, but also adds extra performance overhead.

Keep in mind that NFSv3 *can* use Kerberos, but only the NFS portion of the protocol will use it. Ancillary protocols like mountd, portmapper, NLM, etc. will still be unencrypted. The most secure version of NFS available is NFS v4.x, which combines all the NFS ancillary protocols into a single port and compound NFS calls, which can all be Kerberized.

krb5 – Encrypts the authentication only

Remember when I said if a bad actor stole information about the client, user, etc. that they could spoof that user to get access? Well, with krb5, that becomes harder to do. You can have all the information about the NFS export, but to gain access to a Kerberized mount, you’d need to have a valid username and password to get a Kerberos TGT, which would then be used to get the NFS service ticket. In addition, your client would also need to have a Kerberos ticket and keytab to gain access, so unless your attacker has KDC admin rights, they won’t be able to access the mount.

krb5i – Authentication with data checksums/integrity checking

Krb5 secures the initial authentication for NFS, but the actual NFS data packets are still transmitted in clear text across the wire. If a bad actor were to plug in and sniff those data packets in flight, they could see everything. In addition, bad actors could also act as a “man in the middle” to intercept packets and then transmit their own data if they chose. Krb5i prevents this by using checksums on the client and server to verify that all the data arriving and leaving is coming from a trusted source. The data is still visible, but it can’t be interfered with.

krb5p – Authentication/integrity checking/end to end encryption (privacy)

For the ultimate in-flight security hammer for NFS, you would use krb5p. This combines what krb5i does with in-flight encryption of all NFS packets. That means all NFS packets will be encrypted with the enctype specified in the Kerberos configuration. This can add considerable overhead for NFS performance, but will also prevent NFS packets from being sniffed easily.

How to set up NFS Kerberos in ONTAP

Setting up Kerberos in ONTAP requires a few things:

  • A Key Distribution Center (Microsoft AD, MIT Kerberos, FreeIPA, RedHat IDM)
  • DNS server or static host entries
  • Optional (but recommended): LDAP server for UNIX identity management
  • Configuring NFS clients and ONTAP Service Principal Names

The KDC is the central hub for Kerberos operations and is responsible for handing out Kerberos tickets to clients, users and services for authentication in a Kerberos realm.

DNS is used to resolve IP addresses to host names, which are then passed to the KDC as SPN requests. For example, if client 10.10.10.10 tries to get a Kerberos ticket, a DNS reverse lookup will be done to find the hostname FQDN “client1.domain.com” Then a SPN request for host/client1.domain.com@DOMAIN.COM is made to the KDC. If the SPN exists and matches the request, then the Kerberos request moves on to the next steps. If it doesn’t exist, the request fails.

LDAP is used to centralize the UNIX identities for users and groups to ensure clients and servers have the same information all the time, without manual intervention. This is less important to Kerberos and more important to NFSv4.x and NFS permission resolution.

NFS client configuration for Kerberos on newer clients is fairly straightforward; you configure DNS (so it can find the Kerberos realm name and client hostname) and then simply use one of the automated tools to “join” the Kerberos realm. This automatically creates the service principal and transfers the keytab files. Where it gets tricky is if you have to do a manual Kerberos configuration. TR-4073 and TR-4616 can be of some guidance there, as well as a bevy of articles across the web.

NFS server/ONTAP configuration for Kerberos is relatively simple; you configure DNS and the Kerberos realm and then you enable Kerberos on a network interface. When you do that, ONTAP interacts with the KDC you specified in the Kerberos realm and automates the principal creation – provided the KDC is using standard Kerberos interaction, such as Microsoft Active Directory or kadmin for Linux.

Now we’re getting to the part the title hinted about… FreeIPA Kerberos setup.

Why do I need a blog on FreeIPA Kerberos with ONTAP? You just said it was easy!

I did say it was easy – if the KDC is using standard kadmin. However, FreeIPA happens to use a wrapper over kadmin for KDC management and discourages the use of kadmin for management of service principals. In fact, by default, they lock things down pretty tight.

In FreeIPA, there is a GUI to make things simpler to use. There are also “ipa” commands to interact via the CLI, such as ipa user-add or ipa service-add. In most Linux KDCs, there was kadmin to manage things. In fact, there are *two* kadmins. One is for local management (kadmin.local) and the other is for remote management (kadmin). For more info on the differences, see:

https://docs.oracle.com/cd/E19683-01/816-0211/6m6nc66tj/index.html

The remote kadmin is what ONTAP interacts with for Linux KDCs. When you use the automated ONTAP method to add the NFS service principal, the following happens:

  • A prompt for a username and password to issue a kinit to the KDC to gain access to kadmin
  • get_principal is run via remote kadmin to see if the SPN exists
    • If the SPN doesn’t exist, create_principal is run via remote kadmin
    • If the SPN does exist, modify_principal is run via remote kadmin

All of these kadmin commands require the proper privileges to run successfully. In FreeIPA, remote kadmin is locked down by default to even run get_principal.

For example:

# ipa user-add krbadmin

---------------------
Added user "krbadmin"
---------------------
User login: krbadmin
First name: Kerberos
Last name: Admin
Full name: Kerberos Admin
Display name: Kerberos Admin
Initials: KA
Home directory: /home/krbadmin
GECOS: Kerberos Admin
Login shell: /bin/sh
Principal name: krbadmin@CENTOS-LDAP.LOCAL
Principal alias: krbadmin@CENTOS-LDAP.LOCAL
Email address: krbadmin@centos-ldap.local
UID: 1971600007
GID: 1971600007
Password: False
Member of groups: ipausers
Kerberos keys available: False

# kadmin -p krbadmin
Authenticating as principal krbadmin with password.
Password for krbadmin@CENTOS-LDAP.LOCAL:
kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Operation requires ``get'' privilege while retrieving "nfs/server.domain.com@DOMAIN.COM".

The ONTAP error we’d see is less clear, however. The error suggests that the SPN *already exists*.

Error: NFS Kerberos bind SPN procedure failed
  [  0 ms] Creating account in Unix KDC
  [    30] Successfully connected to ip 10.x.x.x, port 749
           using TCP
**[    40] FAILURE: Unexpected state: Error 1142 at
**         file:src/utils/secd_kadmin_utils.cpp
**         func:createVifKrbAccountUsingKadmin line:227
**[    40] FAILURE: spn already exists. Failed to reuse spn
**         'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
**         admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
**         Unknown code 0
  [    41] Uncaptured failure while creating account

Obviously, the error should refer to some permissions issue. When we see this error in ONTAP,  we can verify on the KDC what is happening via the /var/log/kadmind.log file. In this case, we see “unauthorized request.”

Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Request: kadm5_init, kadmin/admin@CENTOS-LDAP.LOCAL, success, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x, vers=2, flavor=6
Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Unauthorized request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

To give permissions to users to use remote kadmin, you have to modify the /var/kerberos/krb5kdc/kadm5.acl file.

In my setup, this is how I configured the ACL:

# cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@CENTOS-LDAP.LOCAL *
ontap@CENTOS-LDAP.LOCAL *
krbadmin@CENTOS-LDAP.LOCAL *

Then you restart IPA services:

# service ipa restart
Redirecting to /bin/systemctl restart ipa.service

Now, my user can query the principal:

kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Principal does not exist while retrieving "nfs/server.domain.com@DOMAIN.COM".

However, ONTAP gets a new (less descriptive) error when trying to interact with the KDC:

Error: NFS Kerberos bind SPN procedure failed
[ 1 ms] Creating account in Unix KDC
[ 29] Successfully connected to ip x.x.x.x, port 749
using TCP
**[ 45] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:createVifKrbAccountUsingKadmin line:223
**[ 45] FAILURE: Failed to create spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
** Invalid argument
[ 45] Uncaptured failure while creating account

The /var/log/kadmind.log is also fairly useless here:

Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Principal does not exist, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x
Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_create_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid argument, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

ONTAP is just reporting what kadmin says!

Our clue comes from when we try to manually create the SPN using kadmin:

kadmin: add_principal -randkey nfs/server.domain.com@DOMAIN.COM
WARNING: no policy specified for nfs/server.domain.com@DOMAIN.COM; defaulting to no policy
add_principal: Kerberos database constraints violated while creating "nfs/server.domain.com@DOMAIN.COM".

Same goes for modify_principal:

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
(vserver nfs kerberos interface enable)

Username: ontap

Password:

Warning: An account that matches the given service principal name already exists. Re-using this account deletes and re-creates the account if it is not shared by the LIFs in Vserver "NFS". This
invalidates any existing Kerberos service tickets and keys for this account. Do you want to re-use this account? {y|n}: y

Error: NFS Kerberos bind SPN procedure failed
[ 0 ms] Creating account in Unix KDC
[ 39] Successfully connected to ip x.x.x.x, port 749
using TCP
[ 45] Re-using the account for
spn=nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
**[ 53] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:randomizePasswordUsingKadmin line:287
**[ 53] FAILURE: Failed to randomize password for spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'ontap@CENTOS-LDAP.LOCAL'
[ 54] Uncaptured failure while creating account


Mar 24 11:14:54 centos8-ipa.centos-ldap.local kadmind[17208](Notice): Request: kadm5_randkey_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid key/salt tuples, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

And delete_principal:

::*> kerberos interface disable -vserver NFS -lif ipa-krb
(vserver nfs kerberos interface disable)

Username: ontap

Password:

Warning: This command deletes the service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" from the machine account on the KDC. Do you want to continue? {y|n}: y

Error: command failed: Failed to disable NFS Kerberos on LIF "ipa-krb". Failed to delete the account associated with the Kerberos service principal name. Reason: cifs smb kadmin error.

So that means, even though we gave full rights to that user in the kadm5.acl file, we can’t create principals in FreeIPA using kadmin. They want to funnel you to use the IPA tools – likely for a good reason. There’s probably a bunch of other automated steps that take place with those tools, so this is in the interest of simplicity. and security.

How do we get past this?

There are two options here.

  1. Open up the permissions on FreeIPA to use kadmin to create/add/modify/delete principals. (I haven’t figured out how to do this yet)
  2. Manually create the SPN and keytab file and then use ONTAP to transfer the keytab via URI

I went with option 2.

The TL;DR steps, as provided by Alexander Bokovoy in the comments:

- kinit admin
- ipa host-add demo-ipa.centos-ldap.local
- ipa service-add nfs/demo-ipa.centos-ldap.local
- ipa-getkeytab -p nfs/demo-ipa.centos-ldap.local -k ./nfs.keytab -e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

Remember that ONTAP only supports the following enctypes for NFS Kerberos:

  • AES-128 and AES-256
  • DES and 3DES

Then, you copy that file to your HTTP or FTP server. The address to the file will be used in the ONTAP CLI command.

Note: If you’re using IIS, you may have to allow .keytab as a MIME type.

For example:

iis-mime-keytab

Once the file is web-accessible, you run the kerberos interface enable command and use the -keytab-uri option to upload the keytab. Unsupported enctypes will get discarded.

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL -keytab-uri http://web-server/files/ipakrb-ontap.keytab
(vserver nfs kerberos interface enable)

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

Warning: Keys for encryption types "des-cbc-crc,des3-cbc-sha1,aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96" are required for Vserver "NFS" but found keys only for encryption types
"aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96". Keys for encryption types "des-cbc-crc,des3-cbc-sha1" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" are
missing. Available keys will be imported. Do you want to continue? {y|n}: y

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

You’ll also want to ensure you have a valid UNIX user or name mapping rule for the krb-spn name mapping for the NFS clients. That’s covered in TR-4616 in detail.

Once that’s all done, NFS Kerberos mounts should work fine!

[root@centos8-1 sysconfig]# mount -o nfsvers=4,minorversion=1,sec=krb5 x.x.x.x:/kerberos /mnt
[root@centos8-1 sysconfig]# mount | grep krb
x.x.x.x:/kerberos on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.y,local_lock=none,addr=x.x.x.x)
# su ipa-user
sh-4.4$ cd /mnt
sh: cd: /mnt: Permission denied
sh-4.4$ kinit
Password for ipa-user@CENTOS-LDAP.LOCAL:
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ cd /mnt
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
03/23/2020 14:40:16 03/24/2020 14:39:27 nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ touch krb5file
sh-4.4$ ls -la
total 4
drwxrwxrwx. 2 root root 4096 Mar 23 14:40 .
dr-xr-xr-x. 17 root root 224 Jan 21 10:03 ..
-rw-------. 1 ipa-user admins 0 Mar 23 14:40 krb5file
-rw-r--r--. 1 root root 0 Mar 23 14:15 v3nokrb
-rw-r--r--. 1 root root 0 Mar 23 14:16 v4nokrb
-rw-------. 1 ipa-user admins 0 Mar 23 14:22 v4nokrb-ipa

I opened a bug to make this easier in ONTAP with FreeIPA (1308667), so if you run into this, open a case and have it attached to the bug.

If you have questions, suggestions or get stuck (or if I got something wrong), comment below!

ONTAP 9.7 Feature Sneak Peek: NFS client to volume mapping

Since ONTAP was moved from 7-Mode to the more current clustered version, people have been asking for one specific feature:

The ability to map NFS clients to specific volumes and IP addresses.

We’ve had the ability to determine which clients are accessing IP addresses in a cluster (network connections active show), as well as CIFS/SMB session information (cifs session show), but could never get granular information about NFS.

Starting in ONTAP 9.7, we can.

Image result for homer woohoo

The use cases are varied, but usually fall into:

  • Need to discover who’s using a volume before performing migrations, cutovers, etc.
  • Troubleshooting issues
  • Load distribution

In my Unstructured NAS session at Insight 2019, I unveiled the new command that you can use to find NFS client to volume mapping:

cluster::> nfs connected-clients show ?
  [ -instance | -fields <fieldname>, ... ]
  [[-node] <nodename>]                                               Node Name
  [[-vserver] <vserver>]                                             Vserver
  [[-data-lif-ip] <IP Address>]                                      Data LIF IP Address
  [[-client-ip] <IP Address>]                                        Client IP Address
  [[-volume] <volume name>]                                          Volume Accessed
  [[-protocol] <Client Access Protocol>]                             Protocol Version
  [ -idle-time <[<integer>d][<integer>h][<integer>m][<integer>s]> ]  Idle Time (Sec)
  [ -local-reqs <integer> ]                                          Number of Local Reqs
  [ -remote-reqs <integer> ]                                         Number of Remote Reqs

This admin-level command will give you information on which clients are attached to which volume. Plus, if you’re using FlexGroup volumes, you get info all the way down to the member volume level.

For example:

cluster::> nfs connected-clients show -node node1 -vserver DEMO

Node: node1
Vserver: DEMO
Data-Ip: 10.193.67.219
Client-Ip Volume-Name Protocol Idle-Time Local-Reqs Remote-Reqs
--------------- ---------------- -------- ------------- ---------- -----------
10.62.1.173 Tech_ONTAP__0001 nfs3 13h 47m 22s 118 0
10.193.67.121 FGveeam2__0001 nfs3 23h 46m 2s 14326 0
10.193.67.121 FGveeam2__0002 nfs3 1d 8h 34m 0s 0 80118
10.193.67.121 FGveeam4__0001 nfs3 6h 34m 49s 15508 0
10.193.67.121 FGveeam4__0002 nfs3 23h 20m 39s 0 10246
10.193.67.121 FGveeam4__0003 nfs3 23h 20m 35s 10253 0
10.193.67.121 FGveeam4__0004 nfs3 23h 20m 38s 0 5139

(You perhaps notice the FlexGroup name above includes Veeam… that’s a story for another day. :-D)

Enjoy!

*ONTAP 9.7 is due out soon, so stay on the lookout for a blog post announcing its release!

Behind the Scenes: Episode 165 – Accelerate your NAS Data with FlexCache

Welcome to the Episode 165, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we talk about the new iteration of ONTAP’s NAS acceleration feature, FlexCache! Join us as we discuss with NetApp’s Technical Director Pranoop Erasani (pranoop@netapp.com), FlexCache PM Shriya Paramkusam (shriya@netapp.com) and FlexCache TME Chris Hurley (@averageguyx).

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

ONTAP 9.3 NFS sneak preview: Mount and security tracing

aid1871175-v4-728px-trace-step-6-version-2

ONTAP 9.3 is on its way, and with it comes some long-awaited new functionality for NFS debugging, including a way to map volumes to IP addresses!

Mount trace

In ONTAP 7-Mode, you could trace mount requests with an option “nfs.mountd.trace.” That didn’t make its way into ONTAP operating in cluster mode until ONTAP 9.3. I covered a long and convoluted workaround in How to trace NFSv3 mount failures in clustered Data ONTAP.

Now, you can set different levels of debugging for mount traces via the cluster CLI without having to jump through hoops. As a bonus, you can see which data LIF has mounted to which client, to which volume!

To enable it, you would use the following diag level commands:

::*> debug sktrace tracepoint modify -node [node] -module MntTrace -level [0-20] -enabled true

::*> debug sktrace tracepoint modify -node [node] -module MntDebug -level [0-20] -enabled true

When enabled, ONTAP will log the mount trace modules to the sktrace log file, which is located at /mroot/etc/mlog/skrace.log. This file can be accessed via systemshell, or via the SPI interface. Here are a few of the logging levels:

4 – Info
5 – Error
8 – Debug

When you set the trace level to 8, you can see successful mounts, as well as failures. This gives volume info, client IP and data LIF IP. For example, this mount was done from client 10.63.150.161 to data LIF 10.193.67.218 of vserverID 10 on the /FGlocal path:

cluster::*> debug log sktrace show -node node2 -module-level MntTrace_8
Time TSC CPU:INT Module_Level
--------------------- ------------------------ ------- -------------------
 LogMountTrace: Mount access granted for Client=10.63.150.161
 VserverID=10 Lif=10.193.67.218 Path=/FGlocal

With that info, we can run the following command on the cluster to find the SVM and volume:

::*> net int show -address 10.193.67.218 -fields lif
 (network interface show)
vserver lif
------- ---------
DEMO    10g_data1

::*> volume show -junction-path /FGlocal -fields volume
vserver volume
------- ---------------
DEMO    flexgroup_local

The mount trace command can also be used to figure out why mount failures may have occurred from clients. We can also leverage performance information from OnCommand Performance Manager (top clients) and per-client stats to see what volumes might be seeing large performance increases and work our way backward to see what clients are mounting what LIFs, nodes, volumes, etc. with mount trace enabled.

Security trace (sectrace)

In ONTAP 9.2 and prior, you could trace CIFS/SMB permission issues only, using “sectrace” commands. Starting in ONTAP 9.3, you can now use sectrace on SMB and/or NFS. This is useful to troubleshoot why someone might be having access to a file or folder inside of a volume.

With the command, you can filter on:

  • Client IP
  • Path
  • Windows or UNIX name

Currently, sectrace is not supported on FlexGroup volumes, however.

cluster::*> sectrace filter create -vserver DEMO -index 1 -protocols nfs -trace-allow yes -enabled enabled -time-enabled 60

Warning: Security tracing for NFS will not be done for the following FlexGroups because Security tracing for NFS is not supported for FlexGroups: TechONTAP,flexgroupDS,flexgroup_16,flexgroup_local.
Do you want to continue? {y|n}: y

Then, I tested a permission issue.

# mkdir testsec
# chown 1301 testsec/
# chmod 700 testsec
# su user
$ cd /mnt/flexvol/testsec
bash: cd: /mnt/flexvol/testsec: Permission denied

And this was the result:

cluster::*> sectrace trace-result show -vserver DEMO

Node            Index Filter Details             Reason
--------------- ----- -------------------------- ------------------------------
node2           1     Security Style: UNIX       Access is allowed because the
                      permissions                user has UNIX root privileges
                                                 while creating the directory.
                                                 Access is granted for:
                                                 "Append"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is allowed because the
                      permissions                user has UNIX root privileges
                                                 while setting attributes.
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is allowed because the
                                                 permissions user has UNIX root privileges
                                                 while setting attributes.
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is not granted for:
                      permissions                "Modify", "Extend", "Delete"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /
                      Win-User: -
                      UNIX-User: 7041
                      Session-ID: -
node2           1     Security Style: UNIX       Access is not granted for:
                      permissions                "Lookup", "Modify", "Extend",
                                                 "Delete", "Read"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 7041
                      Session-ID: -

As you can see above, the trace output gives a very clear picture about who tried to access the folder, which folder had the error and why the permission issued occurred.

Bonus Round: Block Size Histograms!

Now, this isn’t really a “new in ONTAP 9.3” thing; in fact, I found it as far back as 9.1. I just hadn’t ever noticed it before. But in ONTAP, you can see the block sizes for NFS and CIFS/SMB operations in the CLI with the following command:

cluster::> statistics-v1 protocol-request-size show -node nodename

When you run this, you’ll see the average request size, the total count and a breakdown of what block sizes are being written to the cluster node. This can help you understand your NAS workloads better.

For example, this node runs mostly a VMware datastore workload

cluster::> statistics-v1 protocol-request-size show -node node2 -stat-type nfs3_read

Node: node2
Stat Type: nfs3_read
                     Value    Delta
--------------       -------- ----------
Average Size:        30073    -
Total Request Count: 92633    -
0-511:                1950    -
512-1023:                0    -
1K-2047:              1786    -
2K-4095:              1253    -
4K-8191:             18126    -
8K-16383:              268    -
16K-32767:            4412    -
32K-65535:             343    -
64K-131071:           1560    -
128K - :             62935    -

When you run the command again, you get a delta from the last time you ran it.

If you’re interested in more ONTAP 9.3 feature information, check out Jeff Baxter’s blog here:

https://blog.netapp.com/announcing-netapp-ontap-9-3-the-next-step-in-modernizing-your-data-management/

You can also see me dress up all fancy and break down the new features at a high level here:

I’ll also be doing more detailed blogs on new features as we get closer to the release.

NetApp FlexGroup: Crazy fast

This week, the SPEC SFS®2014_swbuild test results for NetApp FlexGroup volumes submitted for file services were approved and published.

TL;DR – NetApp was the cream of the crop.

You can find those results here:

http://spec.org/sfs2014/results/res2017q3/sfs2014-20170908-00021.html

The testing rig was as follows:

  • Four node FAS8200 cluster (not AFF)
  • 72 4TB 7200 RPM 12Gb SAS drives (per HA pair)
  • NFSv3
  • 20 IBM servers/clients
  • 10GbE network (four connections per HA pair)

Below is a graph that consolidates the results of multiple vendor SPEC SFS®2014_swbuild results. Notice the FlexGroup did more IOPS (around 260k) at a lower latency (sub 3ms):

specsfs-fg

In addition, NetApp had the best Overall Response Time (ORT) of the competition:

specsfs-ort

And had the best MBps/throughput:

specsfs-mbps

Full results here:

http://spec.org/sfs2014/results/sfs2014swbuild.html

For more information on the SPEC SFS®2014_swbuild test, see https://www.spec.org/sfs2014/.

Everything but the kitchen sink…

With a NetApp FlexGroup, the more clients and work you throw at it, the better it will perform. An example of this is seen in TR-4571, with a 2 node A700 doing GIT workload testing. Note how increasing the jobs only encourages the FlexGroup.

average-iops

max-mbps-git

FlexGroup Resources

If you’re interested in learning more, see the following resources:

You can also email us at flexgroups-info@netapp.com.

Tech ONTAP Podcast: Now powered by NetApp FlexGroup volumes!

If you’re not aware, I co-host the Tech ONTAP Podcast. I also am the TME for NetApp FlexGroup volumes. Inexplicably, we weren’t actually storing our podcast files on NetApp storage – instead, we were using the local Mac SSD, which was problematic for three reasons:

  1. It was eventually going to fill up.
  2. If it failed, bye bye files.
  3. It was close to impossible to access unless were were local to the Mac, for a variety of reasons.

So, it finally dawned on me that I had an AFF8040 in my lab, barely being used for anything except testing and TR writing.

At first, I was going to use a FlexVol, out of habit. But then I realized that a FlexGroup volume would provide a great place to write a bunch of 1-400MB files while leveraging all of my cluster resources. The whole process, from creating the FlexGroup, googling autofs in Mac and setting up the NFS mount and Audio Hijack, took me all of maybe 30 minutes (most of that googling and setting up autofs). Not bad!

The podcast setup

When we record the podcast, we use software called Audio Hijack. This allows us to pipe in sound from applications like WebEx and web browsers, as well as from the in-studio microphones, which all get converted to MP3. This is where the FlexGroup NFS mount comes in – we’ll be pointing Audio Hijack to the FlexGroup volume, where the MP3 files will stream in real time.

Additionally, I also migrated all the existing data over to the FlexGroup for archival purposes. We do use OneDrive to do podcast sharing and such, but I wanted an extra layer of centralized data access, and the NFS mounted FlexGroup provides that. Setting it up to stream right from Audio Hijack removes an extra step for me when processing the files. But, before I could point the software at the NFS mount, I had to configure the Mac to automount the FlexGroup volume on boot.

Creating the FlexGroup volume

Normally, a FlexGroup volume is created with 8 member volumes per node for an AFF (as per best practice). However, my FlexGroup volume was going to be around 5TB. That means 16 member volumes would be around 350-400GB each. That violates the other best practices of no less than 500GB per member, to avoid too much remote allocation. While my file sizes weren’t going to be huge, I wanted to avoid issues as the volume filled, so I met in the middle – 8 member volumes total, 4 per node. To do that, you have to go to the CLI; System Manager doesn’t do customization like that yet. In particular, you need the -aggr-list and -aggr-list-multiplier options with volume create.

ontap9-tme-8040::*> vol create -vserver DEMO -volume TechONTAP -aggr-list aggr1_node1,aggr1_node2 -aggr-list-multiplier 4
ontap9-tme-8040::*> vol show -vserver DEMO -volume TechONTAP* -sort-by size -fields size,node
vserver volume size node
------- --------------- ----- ------------------
DEMO TechONTAP__0001 640GB ontap9-tme-8040-01
DEMO TechONTAP__0002 640GB ontap9-tme-8040-02
DEMO TechONTAP__0003 640GB ontap9-tme-8040-01
DEMO TechONTAP__0004 640GB ontap9-tme-8040-02
DEMO TechONTAP__0005 640GB ontap9-tme-8040-01
DEMO TechONTAP__0006 640GB ontap9-tme-8040-02
DEMO TechONTAP__0007 640GB ontap9-tme-8040-01
DEMO TechONTAP__0008 640GB ontap9-tme-8040-02
DEMO TechONTAP 5TB -

Automounting NFS on boot with a Mac

When you mount NFS with a Mac, it doesn’t retain it after you reboot. To get the mount to come back up, you have to configure the autofs service on the Mac. This is different from Linux, where you can simply edit the fstab file. The process is covered very well in this blog post (just be sure to read all the way down to avoid the issue he mentions at the end):

https://coderwall.com/p/fuoa-g/automounting-nfs-share-in-os-x-into-volumes

Here’s my configuration…. I disabled “nobrowse” to prevent issues in case Audio Hijack needed to be able to browse.

autofs.conf

Screen Shot 2017-09-22 at 10.04.37 AM

auto_master file

Screen Shot 2017-09-22 at 10.04.59 AM

auto_nfs

Screen Shot 2017-09-22 at 10.05.17 AM

After that was set up, I copied over the existing 50-ish GBs of data into the FlexGroup and cleaned up some space on the Mac.

ontap9-tme-8040::*> vol show -vserver DEMO -volume TechONTAP* -sort-by size -fields size,used
vserver volume size used
------- --------------- ----- -------
DEMO TechONTAP__0001 640GB 5.69GB
DEMO TechONTAP__0002 640GB 8.24GB
DEMO TechONTAP__0003 640GB 5.56GB
DEMO TechONTAP__0004 640GB 6.48GB
DEMO TechONTAP__0005 640GB 6.42GB
DEMO TechONTAP__0006 640GB 8.39GB
DEMO TechONTAP__0007 640GB 6.25GB
DEMO TechONTAP__0008 640GB 6.25GB
DEMO TechONTAP 5TB 53.29GB
9 entries were displayed.

Then, I configured Audio Hijack to pump the recordings to the FlexGroup volume.

Screen Shot 2017-09-22 at 10.01.00 AM.png

Then, we recorded a couple episodes, without an issue!

Screen Shot 2017-09-22 at 10.34.30 AM.png

As you can see from this output, the FlexGroup volume is relatively evenly allocated:

ontap9-tme-8040::*> node run * flexgroup show TechONTAP
2 entries were acted on.

Node: ontap9-tme-8040-01
FlexGroup 0x80F03817
* next snapshot cleanup due in 2886 msec
* next refresh message due in 886 msec (last to member 0x80F0381F)
* spinnp version negotiated as 4.6, capability 0x3
* Ref count is 8

Idx Member L Used Avail Urgc Targ Probabilities D-Ingest Alloc F-Ingest Alloc
--- -------- - --------------- ---------- ---- ---- --------------------- --------- ----- --------- -----
 1 2044 L 1485146 0% 159376256 0% 12% [100% 100% 79% 79%] 0+ 0 0 0+ 0 0
 2 2045 R 2153941 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 3 2046 L 1415120 0% 159339950 0% 12% [100% 100% 76% 76%] 0+ 0 0 0+ 0 0
 4 2047 R 1690392 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 5 2048 L 1675583 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 6 2049 R 2191360 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 7 2050 L 1630946 1% 159376256 0% 12% [100% 100% 87% 87%] 0+ 0 0 0+ 0 0
 8 2051 R 1631429 1% 159376256 0% 12% [100% 100% 87% 87%] 0+ 0 0 0+ 0 0

Node: ontap9-tme-8040-02
FlexGroup 0x80F03817
* next snapshot cleanup due in 3144 msec
* next refresh message due in 144 msec (last to member 0x80F03818)
* spinnp version negotiated as 4.6, capability 0x3
* Ref count is 8

Idx Member L Used Avail Urgc Targ Probabilities D-Ingest Alloc F-Ingest Alloc
--- -------- - --------------- ---------- ---- ---- --------------------- --------- ----- --------- -----
 1 2044 R 1485146 0% 159376256 0% 12% [100% 100% 79% 79%] 0+ 0 0 0+ 0 0
 2 2045 L 2153941 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 3 2046 R 1415120 0% 159339950 0% 12% [100% 100% 76% 76%] 0+ 0 0 0+ 0 0
 4 2047 L 1690392 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 5 2048 R 1675583 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 6 2049 L 2191360 1% 159376256 0% 12% [100% 100% 98% 98%] 0+ 0 0 0+ 0 0
 7 2050 R 1630946 1% 159376256 0% 12% [100% 100% 87% 87%] 0+ 0 0 0+ 0 0
 8 2051 L 1631429 1% 159376256 0% 12% [100% 100% 87% 87%] 0+ 0 0 0+ 0 0

I plan on using this setup when I start writing the new FlexGroup data protection best practice guide, so stay tuned for that…

So, now, the Tech ONTAP podcast is happily drinking the NetApp FlexGroup champagne!

If you’re going to NetApp Insight, check out session 16594-2 on FlexGroup volumes.

For more information on NetApp FlexGroup volumes, see:

Using NFSv4.x ACLs with NFSv3 in NetApp ONTAP? You betcha!

One thing I’ve come to realize from being in IT so long is that you should be constantly learning new things. If you aren’t, it’s not because you’re smart or because you know everything; it’s because you’re stagnating.

So, I was not surprised when I heard it was possible to apply NFSv4.x ACLs to files and folders and then mount them via NFSv3 and have the ACLs still work! I already knew that you could do audit ACEs from NFSv4.x for NFSv3 (covered in TR-4067), but had no idea this could extend into the permissions realm. If so, this solves a pretty big problem with NFSv3 in general, where your normal permissions are limited only to owner, group and then everyone else. That makes it hard to do any sort of granular access control for NFSv3 mounts, presents problems for some environments.

It also allows you to keep using NFSv3 for your workloads, whether for legacy application or general performance concerns. NFSv4.x has a lot of advantages over NFSv3, but if you don’t need stateful operations or the NFSv4.x features, or integrated locking, then you are safe to stay with NFSv3.

So, is it possible to use NFSv4.x ACLs with NFSv3 objects?

You betcha!

fargo-film-marge.jpg

The method for doing this is pretty straightforward.

  1. Configure and enable NFSv4.x in ONTAP and on your client
  2. Enable NFSv4.x ACL support in ONTAP
  3. Mount the export via NFSv4.x
  4. Apply the NFSv4.x ACLs
  5. Unmount and then remount the export using NFSv3 and test it out!

 

Configuring NFSv4.x

When you’re setting up NFSv4.x in an environment, there are a few things to keep in mind:

  • Client and NFS server support for NFSv4.x
  • NFS utilities installed on clients (for NFSv4.x functionality)
  • NFSv4.x configured on the client in idmapd.conf
  • NFSv4.x configured on the server in ONTAP (ACLS allowed)
  • Export policies and rules configured in ONTAP
  • Ideally, a name service server (like LDAP) to negotiate the server/client conversation of user identities

One of the reasons NFS4.x is more secure than NFSv3 is the use of user ID strings (such as user@domain.com) to help limit cases of user spoofing in NFS conversations. This ID string is required to be case-sensitive. If the string doesn’t match on both client and server, then the NFSv4.x mounts will get squashed to the defined “nobody” user in the NFSv4.x client. One of the more common issues seen with NFSv4.x mounts is the “nobody:nobody” user and group on files and folders. One of the most common causes of this is when a domain string is mismatched on the client and server.

In a client that domain string is defined in the idmapd.conf file. Sometimes, it will default to the DNS domain. In ONTAP, the v4-id-domain string should be configured to the same value on the client to provide proper NFSv4.x authentication.

Other measures, such as Kerberos encryption, can help lock the NFS conversations down further. NFSv4.x ACLs are a way to ensure that files and folders are only seen by those entities that have been granted access and is considered to be authorization, or, what you are allowed to do once you authenticate. For more complete steps on setting up NFSv4.x, see TR-4067 and TR-4073.

However, we’re only setting up NFSv4.x to allow us to configure the ACLs…

What are NFSv4.x ACLs?

NFSv4.x ACLs are a way to apply granular permissions to files and folders in NFS outside of the normal “read/write/execute” of NFSv3, and across more objects than simple “owner/group/everyone.” NFSv4.x ACLs allow administrators to set permissions for multiple users and groups on the same file or folder and treat NFS ACLs more like Windows ACLs. For more information on NFSv4.x ACLs, see:

http://wiki.linux-nfs.org/wiki/index.php/ACLs

https://linux.die.net/man/5/nfs4_acl

http://www.netapp.com/us/media/tr-4067.pdf

NFSv3 doesn’t have this capability by default. The only way to get more granular ACLs in NFSv3 natively is to use POSIX ACLs, which ONTAP doesn’t support.

Once you’ve enabled ACLs in ONTAP (v4.0-acl and/or v4.1-acl options), you can mount an NFS export via NFSv4.x and start applying NFSv4.x ACLs.

In my environment, I mounted a homedir volume and then set up an ACL on a file owned by root for a user called “prof1” using nfs4_setfacl -e (which allows you to edit a file rather than have to type in a long command).

[root@centos7 /]# mount demo:/home /mnt
[root@centos7 /]# mount | grep mnt
demo:/home on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.193.67.225,local_lock=none,addr=10.193.67.237)

The file lives in the root user’s homedir. The root homedir is set to 755, which means anyone can read them, but no one but the owner (root) can write to them.

drwxr-xr-x 2 root root 4096 Jul 13 10:42 root

That is, unless, I set NFSv4.x ACLs to allow a user full control:

[root@centos7 mnt]# nfs4_getfacl /mnt/root/file
A::prof1@ntap.local:rwaxtTnNcCy
A::OWNER@:rwaxtTnNcCy
A:g:GROUP@:rxtncy
A::EVERYONE@:rxtncy

I can also see those permissions from the ONTAP CLI:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: NFSV4 Security Descriptor
 Control:0x8014
 DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 ALLOW-OWNER@-0x1601bf
 ALLOW-GROUP@-0x1200a9-IG
 ALLOW-EVERYONE@-0x1200a9

I can also expand the mask to translate the hex:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file -expand-mask true

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: 0x20
 ...0 .... .... .... = Offline
 .... ..0. .... .... = Sparse
 .... .... 0... .... = Normal
 .... .... ..1. .... = Archive
 .... .... ...0 .... = Directory
 .... .... .... .0.. = System
 .... .... .... ..0. = Hidden
 .... .... .... ...0 = Read Only
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: NFSV4 Security Descriptor
 Control:0x8014

1... .... .... .... = Self Relative
 .0.. .... .... .... = RM Control Valid
 ..0. .... .... .... = SACL Protected
 ...0 .... .... .... = DACL Protected
 .... 0... .... .... = SACL Inherited
 .... .0.. .... .... = DACL Inherited
 .... ..0. .... .... = SACL Inherit Required
 .... ...0 .... .... = DACL Inherit Required
 .... .... ..0. .... = SACL Defaulted
 .... .... ...1 .... = SACL Present
 .... .... .... 0... = DACL Defaulted
 .... .... .... .1.. = DACL Present
 .... .... .... ..0. = Group Defaulted
 .... .... .... ...0 = Owner Defaulted

DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .1.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...1 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...1 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .1.. = Append
 .... .... .... .... .... .... .... ..1. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-OWNER@-0x1601bf
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .1.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...1 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...1 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .1.. = Append
 .... .... .... .... .... .... .... ..1. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-GROUP@-0x1200a9-IG
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .0.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...0 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...0 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .0.. = Append
 .... .... .... .... .... .... .... ..0. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-EVERYONE@-0x1200a9
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .0.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...0 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...0 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .0.. = Append
 .... .... .... .... .... .... .... ..0. = Write
 .... .... .... .... .... .... .... ...1 = Read

In the above, I gave prof1 full control over the file. Then, I mounted via NFSv3:

[root@centos7 /]# mount -o nfsvers=3 demo:/home /mnt
[root@centos7 /]# mount | grep mnt
demo:/home on /mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.193.67.219,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.193.67.219)

When I become a user that isn’t on the NFSv4.x ACL, I can’t write to the file:

[root@centos7 /]# su student1
sh-4.2$ cd /mnt/root
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwxr-xr-x 1 root bin 0 Jul 13 10:23 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

sh-4.2$ touch file
touch: cannot touch ‘file’: Permission denied
sh-4.2$ rm file
rm: remove write-protected regular empty file ‘file’? y
rm: cannot remove ‘file’: Permission denied

When I change to the prof1 user, I have access to do whatever I want, even though the mode bit permissions in v3 say I can’t:

[root@centos7 /]# su prof1
sh-4.2$ cd /mnt/root
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwxr-xr-x 1 root bin 0 Jul 13 10:23 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

sh-4.2$ vi file
sh-4.2$ cat file
NFSv4ACLS!

When I do a chmod, however, nothing seems to change from the NFSv4 ACL for the user. I set 700 on the file, which shows up in NFSv3 mode bits:

sh-4.2$ chmod 700 file
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwx------ 1 root bin 11 Aug 11 09:58 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

But notice how the prof1 user still has full control:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 700
 UNIX Mode Bits in Text: rwx------
 ACLs: NFSV4 Security Descriptor
 Control:0x8014
 DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 ALLOW-OWNER@-0x1601bf
 ALLOW-GROUP@-0x120088-IG
 ALLOW-EVERYONE@-0x120088

This is because of an ONTAP option known as “ACL Preservation.”

ontap9-tme-8040::*> nfs show -vserver DEMO -fields v4-acl-preserve
vserver v4-acl-preserve
------- ---------------
DEMO enabled

When I set the option to enabled, the NFSv4.x ACLs will survive mode bit changes. If I disable the option, the ACLs get blown away when a chmod is done:

ontap9-tme-8040::*> nfs modify -vserver DEMO -v4-acl-preserve disabled

ontap9-tme-8040::*> nfs show -vserver DEMO -fields v4-acl-preserve
vserver v4-acl-preserve
------- ---------------
DEMO disabled


[root@centos7 root]# chmod 755 file

And the ACLs are wiped out:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: -

I’d personally recommend setting that option to “enabled” if you want to do v3 mounts with v4.x ACLs.

So, there you have it… a new way to secure your NFSv3 mounts!

How to trace NFSv3 mount failures in clustered Data ONTAP

One of the questions I get on a fairly regular basis is:

I used to be able to trace NFSv3 mount failures in 7-Mode with an option called nfs.mountd.trace. How do I do that in clustered Data ONTAP?

Well, the short answer is that there is no single option to trace mount failures currently. The NFS export architecture is vastly different in clustered Data ONTAP than in 7-Mode.

However, there is a way to get pretty close to replicating the functionality. And I’ve managed to script it out a bit to simplify the process. The following is the first run of a draft of a new section in TR-4067: NFS Best Practices and Implementation Guide. This blog will provide a sneak preview of that section, which may also become the beginnings of a clustered Data ONTAP NAS troubleshooting guide.

troubleshooting-large

This document contains commands that must be run at diag privilege level. Please exercise caution when running these commands. Contact NetApp Technical Support for guidance.

What is NFSv3 Mount Tracing?

In some instances, NFS mount requests may fail for clients. When this happens, troubleshooting a mount needs to be made as simple as possible. In Data ONTAP operating in 7-Mode, there was an option that allowed storage administrators to toggle for NFSv3 mount troubleshooting:

nfs.mountd.trace
Note: NFSv4 mounts do not leverage the MOUNT protocol, so these steps do not apply to NFSv4.

When the above option is enabled in 7-Mode, all mount requests are logged. This option is intended to help debug denied mount requests. Valid values for this option are on (enabled) or off (disabled). The default value for this option is off to avoid too many messages. The logging output is stored in the /etc/messages file, as well as being piped to the CLI console.

When a successful mount occurred on a NFS server in 7-Mode, something similar to the following would have been seen in the messages file:

Thu Feb 25 17:03:28 GMT: Client 10.61.69.167 (xid 1456151530), is sent the NULL reply
Thu Feb 25 17:03:28 GMT: Client 10.61.69.167 (xid 4006546062), is sent the NULL reply
Thu Feb 25 17:03:28 GMT: Client 10.61.69.167 (xid 4023323278) in mount, has access rights to path

When a failed mount occurred, we’d see something like this:

Thu Feb 25 17:07:13 GMT: Client 10.61.69.167 (xid 945700734), is sent the NULL reply
Thu Feb 25 17:07:13 GMT: Client 10.61.69.167 (xid 962477950) in mount, does not have access rights to path /vol/nfs

Essentially, the information given in the trace included:

  • Timestamp of error logged to console and messages file
  • Client IP address
  • Whether the client received a reply
  • Whether the mount succeeded or not

That was the extent of the information given. From there, a storage administrator could use a command like exportfs -c to check access, which would give a more descriptive error:

fas2020-rtp2*> exportfs -c 10.61.69.167 /vol/nfs
exportfs: problem finding mount access for 10.61.69.167 to /vol/nfs: (Export not there)

In the above example, the volume “nfs” was in the exports file, but that file had not been applied to memory yet (hence, “export not there”).

When we review the in-memory exports vs. the exports file, that is confirmed.

fas2020-rtp2*> exportfs
/vol/vol0       -sec=sys,rw,root=10.61.69.161,nosuid

fas2020-rtp2*> rdfile /etc/exports
/vol/vol0       -sec=sys,rw,root=10.61.69.161,nosuid
/vol/nfs        -sec=sys,rw,nosuid

All it took to fix this issue was re-exporting:

fas2020-rtp2*> exportfs -a

fas2020-rtp2*> exportfs -c 10.61.69.167 /vol/nfs
exportfs: 10.61.69.167 has mount access to /vol/nfs

This is what a client export access issue would look like:

fas2020-rtp2*> exportfs
/vol/vol0       -sec=sys,rw,root=10.61.69.161,nosuid
/vol/nfs        -sec=sys,rw,nosuid
/vol/nfs2       -sec=sys,ro=10.61.69.166,rw=10.61.69.16,nosuid

Thu Feb 25 17:15:34 GMT: Client 10.61.69.167 (xid 1456395643), is sent the NULL reply
Thu Feb 25 17:15:34 GMT: Client 10.61.69.167 (xid 2136546296), is sent the NULL reply
Thu Feb 25 17:15:34 GMT: Client 10.61.69.167 (xid 2153323512) in mount, does not have access rights to path /vol/nfs2

fas2020-rtp2*> exportfs -c 10.61.69.167 /vol/nfs2
exportfs: 10.61.69.167 does not have mount access to /vol/nfs2 (Access denied)

With the option nfs.mountd.trace, the “who, what and when” of the mount issue was tracked, but not the “how or why.” Those questions required additional leg work.

Despite the limitations of nfs.mountd.trace, customers who have moved from Data ONTAP operating in 7-Mode to clustered Data ONTAP miss this option, as it fit into a specific workflow. However, we can replicate the data gathered by this option fairly closely in clustered Data ONTAP. The following sections show how.

Replicating the option in clustered Data ONTAP

nfs.mountd.trace is conspicuously absent in clustered Data ONTAP. However, that doesn’t mean storage administrators can’t still get the same information as before. It just takes a different approach.

Some of this approach is currently covered in the following KB article: https://kb.netapp.com/support/index?page=content&id=2017281

That KB shows a number of statistics that can help indicate there is an issue with exports/NFS access. It discusses going into systemshell, but we can actually gather statistics from the cluster shell.

Using NFS Statistics to troubleshoot NFS mount issues

Using NFS export access statistics, I can check to see if I’ve ever been denied access to a mount and if those counters are incrementing.

To collect stats, we have to start the statistics capture for the object nfs_exports_access_cache. The object is only available at diag privilege:

cm8040-cluster::*> set diag 
cm8040-cluster::*> statistics start -object nfs_exports_access_cache 
Statistics collection is being started for sample-id: sample_17602

Keep in mind that in 8.3 and later, trying to view stats before we have started them will result in this error:

cm8040-cluster::*> statistics show -object nfs_exports_access_cache
Error: show failed: Default sample not found. To collect a sample, 
use the "statistics start" and "statistics stop" commands. 
To view the data sample, use the "statistics show" command with the 
"-sample-id" parameter.

 

After a bit, check the stats. Filter on the following counters, using a pipe symbol (|) to indicate there are multiple entries:

cm8040-cluster::*> statistics show -object nfs_exports_access_cache -counter cache_entries_negative|cache_entries_positive|denied|denied_no_eff_rorule|denied_no_rule|denied_no_rorule|denied_wrong_vserver|export_check|indeterminate

Object: nfs_exports_access_cache
Instance: cm8040-cluster-01
Start-time: 2/25/2016 17:35:00
End-time: 2/25/2016 17:44:09
Elapsed-time: 549s

Node: cm8040-cluster-01

    Counter                                                     Value
    -------------------------------- --------------------------------
    cache_entries_negative                                          2
    cache_entries_positive                                          2
    denied                                                          0
    denied_no_eff_rorule                                            0
    denied_no_rorule                                                0
    denied_no_rule                                                  0
    denied_wrong_vserver                                            0
    export_check                                                   92
    indeterminate                                                   0

In the above, I can see that I have two negative entries in cache (which means export access was denied twice). If I know the client’s IP address already (for instance, if the owner of the client called in to complain about being denied access), I could look that information up in the cache using the following command at diag privilege:

cm8040-cluster::*> diag exports nblade access-cache show -node cm8040-cluster-01 -vserver nfs -policy denyhost -address 10.61.69.167

                               Node: cm8040-cluster-01
                            Vserver: nfs
                        Policy Name: denyhost
                         IP Address: 10.61.69.167
           Access Cache Entry Flags: has-usable-data
                        Result Code: 0
                  Failure Type Code: 0
        First Unresolved Rule Index: -
             Unresolved Clientmatch: -
             Unresolved Anon String: -
     Number of Matched Policy Rules: 0
List of Matched Policy Rule Indexes: -
                       Age of Entry: 1002s
        Access Cache Entry Polarity: negative

In the above, the stats were not enabled when the initial issue occurred. Once I reproduce the access issue, I can check the stats again:

cm8040-cluster::*> statistics show -object nfs_exports_access_cache -counter cache_entries_negative|cache_entries_positive|denied|denied_no_eff_rorule|denied_no_rule|denied_no_rorule|denied_wrong_vserver|export_check|indeterminate

Object: nfs_exports_access_cache
Instance: cm8040-cluster-01
Start-time: 2/25/2016 17:35:00
End-time: 2/25/2016 17:50:43
Elapsed-time: 943s

Node: cm8040-cluster-01

    Counter                                                     Value
    -------------------------------- --------------------------------
    cache_entries_negative                                          2
    cache_entries_positive                                          2
    denied                                                          1
    denied_no_eff_rorule                                            0
    denied_no_rorule                                                0
    denied_no_rule                                                  1
    denied_wrong_vserver                                            0
    export_check                                                  160
    indeterminate                                                   0

We can see that the denied_no_rule stat incremented for node1. If the policy rule allows the client but simply denies access:

cm8040-cluster::*> export-policy rule show -vserver nfs -policyname denyhost
             Policy          Rule    Access   Client                RO
Vserver      Name            Index   Protocol Match                 Rule
------------ --------------- ------  -------- --------------------- ---------
nfs          denyhost        1       any      10.61.69.166          any
nfs          denyhost        2       any      10.61.69.167          never
2 entries were displayed.

cm8040-cluster::*> statistics show -object nfs_exports_access_cache -counter cache_entries_negative|cache_entries_positive|denied|denied_no_eff_rorule|denied_no_rule|denied_no_rorule|denied_wrong_vserver|export_check|indeterminate

Object: nfs_exports_access_cache
Instance: cm8040-cluster-01
Start-time: 2/25/2016 17:35:00
End-time: 2/25/2016 18:41:47
Elapsed-time: 4007s

Node: cm8040-cluster-01

    Counter                                                     Value
    -------------------------------- --------------------------------
    cache_entries_negative                                          2
    cache_entries_positive                                          2
    denied                                                          2
    denied_no_eff_rorule                                            0
    denied_no_rorule                                                1
    denied_no_rule                                                  1
    denied_wrong_vserver                                            0
    export_check                                                  714
    indeterminate                                                   0

This time, denied_no_rorule increments when a client attempts access, which means the export policy rule specified the client by name or IP and explicitly denied any access. Now, I have the “what” and “why” questions answered: access denied because of no export policy rule for my client, whether by accident or by design.

While statistics are great, these numbers don’t tell the full story. For instance, with just the numbers, we don’t know which clients are having access issues, what volumes are affected, and thus, what specific export policies/rules are the culprit. We do know what node is being accessed, however, which is useful.

Using logs to troubleshoot NFS mount issues

We can use statistics to show us that a problem exists. We use logs to discover where the problem lives.

In clustered Data ONTAP, there are a number of log traces that can be enabled to not only gain the same level of functionality as seen in 7-Mode, but also expands on that level of functionality. The downside is that instead of a single option, there are multiple switches to toggle.

To get a full look at debug logs for the NFS stack, we would need to configure sktraces to debug levels:

sysvar.sktrace.AccessCacheDebug_enable=-1
sysvar.sktrace.NfsPathResolutionDebug_enable=63
sysvar.sktrace.NfsDebug_enable=63
sysvar.sktrace.MntDebug_enable=-1
sysvar.sktrace.Nfs3ProcDebug_enable=63

This can be done via cluster shell (the ::> prompt) rather than dropping into the systemshell. Since we know from our statistics that the issue lives on node1 of the cluster, we can limit the command to node1 using the following command:

cm8040-cluster::*> set diag -c off; systemshell -node cm8040-cluster-01 -c 
"sudo sysctl sysvar.sktrace.AccessCacheDebug_enable=-1;
sudo sysctl sysvar.sktrace.NfsPathResolutionDebug_enable=63;
sudo sysctl sysvar.sktrace.NfsDebug_enable=63;
sudo sysctl sysvar.sktrace.MntDebug_enable=-1;
sudo sysctl sysvar.sktrace.Nfs3ProcDebug_enable=63;
sudo sysctl sysvar.sktrace.NfsDebug_enable=-1"

Export rules are checked via mgwd, so we’d need to log at a debug level there as well:

cm8040-cluster::*> set diag -c off; logger mgwd log modify -module mgwd::exports 
-level debug -node cm8040-cluster-01

Starting in clustered Data ONTAP 8.3.1, disable log supression. If suppression is enabled, we might miss important information in the logs.

NOTE: Not available prior to clustered Data ONTAP 8.3.1.

cm8040-cluster::*> diag exports mgwd journal modify -node cm8040-cluster-01 
-trace-all true -suppress-repeating-errors false

After these are set, reproduce the issue. After reproducing the issue, be sure to disable the debug logging by setting the values back to their defaults to avoid flooding logs with spam, which could cause us to miss errors as the logs roll off.

The default sktrace values are:

sysvar.sktrace.AccessCacheDebug_enable=0
sysvar.sktrace.NfsPathResolutionDebug_enable=0
sysvar.sktrace.NfsDebug_enable=0
sysvar.sktrace.MntDebug_enable=0
sysvar.sktrace.Nfs3ProcDebug_enable=0

To disable sktraces:

cm8040-cluster::*> set diag -c off; systemshell -node cm8040-cluster-01 
-c "sudo sysctl sysvar.sktrace.AccessCacheDebug_enable=0;
sudo sysctl sysvar.sktrace.NfsPathResolutionDebug_enable=0;
sudo sysctl sysvar.sktrace.NfsDebug_enable=0;
sudo sysctl sysvar.sktrace.MntDebug_enable=0;
sudo sysctl sysvar.sktrace.Nfs3ProcDebug_enable=0;
sudo sysctl sysvar.sktrace.NfsDebug_enable=0"

To reset the mgwd exports trace to error level:

cm8040-cluster::*> set diag -c off; logger mgwd log modify -module mgwd::exports 
-level err -node cm8040-cluster-01

To the mgwd journal log and re-enable suppression:

cm8040-cluster::*> diag exports mgwd journal modify -node cm8040-cluster-01 
-trace-all false -suppress-repeating-errors true

Collecting logs

The logs that capture the necessary information are in /mroot/etc/log/mlog on the specific node that is being debugged. These are accessible via the SPI web interface covered in this KB article:

How to manually collect logs and copy files from a clustered Data ONTAP storage system

(TL;DR: Use http://cluster-mgmt-IP/spi)

Specifically, the logs we want are

sktlogd.log
mgwd.log
secd.log

Analyzing the logs

These logs can be a bit difficult to digest if you don’t know what you’re looking for. To make the logs more consumable, find clients that have tried mounting during the log capture using grep (or search via text editor) in the log for the following string:

"MntDebug_3:  MountProcNull: From Client"

For example:

% cat sktlogd.log | grep "MntDebug_3:  MountProcNull: From Client"
000000bb.000635b7 0697ecee Thu Feb 25 2016 18:59:41 +00:00 [kern_sktlogd:info:4513] 22.18.39.700592+1666 [2.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167
000000bb.000635d1 0697ecef Thu Feb 25 2016 18:59:41 +00:00 [kern_sktlogd:info:4513] 22.18.39.701423+0092 [7.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167
000000bb.00066e0b 0698181b Thu Feb 25 2016 19:18:07 +00:00 [kern_sktlogd:info:4513] 22.37.07.606654+0674 [2.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167
000000bb.00066e25 0698181b Thu Feb 25 2016 19:18:07 +00:00 [kern_sktlogd:info:4513] 22.37.07.612221+0020 [7.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167
000000bb.00068db1 06982ad7 Thu Feb 25 2016 19:26:07 +00:00 [kern_sktlogd:info:4513] 22.45.05.424066+0094 [7.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167
000000bb.00068dcb 06982ad7 Thu Feb 25 2016 19:26:07 +00:00 [kern_sktlogd:info:4513] 22.45.05.424789+2000 [7.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167
000000bb.00069500 06982fd2 Thu Feb 25 2016 19:28:14 +00:00 [kern_sktlogd:info:4513] 22.47.12.745115+0720 [7.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167
000000bb.0006951a 06982fd2 Thu Feb 25 2016 19:28:14 +00:00 [kern_sktlogd:info:4513] 22.47.12.745837+0236 [2.0] MntDebug_3:  MountProcNull: From Client = 10.61.69.167

From the above, we get our “when” (time stamp) and “who” (which client) answers. Then we can see the “why” (why did it fail) with the following log sections:

000000bb.0006957e 06982fd3 Thu Feb 25 2016 19:28:14 +00:00 [kern_sktlogd:info:4513] 22.47.12.746272+0446 [7.0] MntDebug_5:  MountProcMntPathResolutionCallback: Export Check Exec=0x0xffffff08b9f4b040,rsid=0,Result=3106
000000bb.0006957f 06982fd3 Thu Feb 25 2016 19:28:14 +00:00 [kern_sktlogd:info:4513] 22.47.12.746273+0380 [7.0] MntDebug_5:  MountProcMntPathResolutionCallback: No Access Exec=0x0xffffff08b9f4b040 Ecode=0xd Result=3106

Security Daemon (SecD) Logs

In addition to the logging that can be done at the kernel layer, there are also logs that can be leveraged with the authentication process. These logs are located at /mroot/etc/log/mlog.

In the SecD log, we’re mainly looking for two things with regards to mount access problems:

  • Is SecD running? (if not, no mounts are allowed)
  • What user is the client authenticating to? (i.e., user squashing)
  • Netgroup processing (no netgroup resolution? Access will be denied.)

Outside of that, SecD logs are mainly useful in troubleshooting user access permissions after the mount is successful, which falls under “permissions” or “access” issues, rather than mounts failing.

For more information on SecD and what it is, see TR-4073: Secure Unified Authentication and TR-4067: NFS Best Practice and Implementation Guide.

Packet tracing

With mount tracing in clustered Data ONTAP, there is currently no way to answer the “what” (ie, which volume is being mounted) from the logs. That would have to happen via a packet trace. Packet traces can be captured from an NFS client using tcpdump. The syntax I normally use to trace from an NFS client:

# tcpdump -w <filename.trc> -s 0 -i <interfacename>

In the above example:

  • If you don’t specify –w, the output will pipe to the screen and won’t get captured to a file
  • -s 0 is used to ensure all of the packetlength is captured and avoids truncation
  • Interface names can be seen with ifconfig. The interface should be the one with the IP address connecting to the NFS server.

Packet traces can also be captured from the cluster. These should be captured simultaneously with the client traces. The goal is to ensure the client and server are communicating properly and consistently. Clustered Data ONTAP packet tracing is covered in the following KB article:

How to capture packet traces on clustered Data ONTAP systems

What to look for in packet traces

When troubleshooting mount access issues via packet traces, it’s useful to filter on NFSv3 specific calls. For example, in Wireshark, use the following filter:

nfs or mount or portmap

That will filter out the main traffic we need to troubleshoot mount issues.

To find export paths, simply look for the MOUNT calls. That will tell us what export is being mounted:

MOUNT  150    V3 MNT Call (Reply In 67) /unix

To find which volume in the cluster that path corresponds to, use the following command:

cluster::> volume show -junction-path /unix

If the export path is a subdirectory or qtree, use the first portion of the path. For example, /unix/qtree/dir1 would be queried as /unix to find the corresponding volume.

Other useful information in packet traces that can be used to troubleshoot mount issues include:

  • Source and destination IP addresses
  • Source and destination ports (firewall issues, NFS/mount rootonly option issues)
  • User attempting access
  • Auth method being used (AUTH_SYS, AUTH_NONE, AUTH_GSS, etc)
  • Client hostname
  • Filehandles

Simplifying NFS mount debugging

To make troubleshooting NFS mounts a bit simpler, it makes sense to create a shell script to automate the enabling/disabling of NFS tracing in clustered Data ONTAP.

First, passwordless SSH should be configured for use with an admin host. That process is covered in TR-4073: Secure Unified Authentication, as well as this KB: https://kb.netapp.com/support/index?page=content&id=1012542

Once that’s done, create a shell script and customize it. And feel free to add feedback or your own submissions!

Script templates are located on Github:

https://github.com/whyistheinternetbroken/TR-4067-scripts

TECH::What Super Bowl 49 Taught Us About Transitioning from NFSv3 to NFSv4.x

I’m sure most everyone either watched Super Bowl 49 or at least read about it somewhere. The Seattle Seahawks and New England Patriots clashed down to the wire for one of the most thrilling games in recent history. Both sides played fairly well, but not without mistakes. And as most of us know, it all came down to a final decision – do we run or pass from the 1 yard line?

The Seahawks chose to pass and the rest was history.

So what lessons can storage administrators learn about NFS from football (or as some call it, sportball)? In this case, we’ll call NFSv3 “the run” and NFSv4.x? The forward pass.

The forward pass: A little history

American football was not always the game it is today. The sport evolved from rugby and the first American football game was played in 1869. Like rugby, it was a run-heavy, violent game. Keep in mind that these used to be the helmets, and helmets weren’t even worn in the early games:

Yeah… that’s not gonna hurt, right? Source: AntiqueAthlete.com

In 1906, a rule change was added to try to help make the game less… bloody.

Mutant League Football – Source: Game Informer

As a result, the game opened up and became more exciting with higher scores and it added a facet of strategy to the game. Now it wasn’t just about simple brute force – you had to try to guess what the other team was going to do based on the situation.

NFS: Not Football Specific

NFS actually stands for Network File System and is used in enterprise environments across the world for file storage. Everything from music to movies to medical images to seismic data to virtual machines are hosted on NFS storage (preferably NetApp :-P). There are plenty of resources on what it is out there, starting with the Request For Comments (RFC) standards. As far as NFS on NetApp, I cover it in some detail in the following technical reports (Light reading, especially if you’re having trouble sleeping.):

But NFS, and NAS storage administration, has some things in common with football – you have to decide on what to do based on every situation.

NFSv4.x and the forward pass

If we look at the decision made last night not to run  the ball on 2nd and goal from the 1 yard line with a player nicknamed “Beast Mode” (how do you NOT run it??) in terms of transitioning from the generally rock-solid NFSv3 to NFSv4.x, we can make the following analogies.

  • Do I give the ball to my best player (NFSv3) that has been dependable for years and see what happens?
    I know it could backfire on me, and he does have his flaws…
  • Do I throw a forward pass (NFSv4.x) without really thinking about the immediate consequences of just jumping into that decision?
    I could misconfigure the pass and that would be it for my enterprise data storage…
  • Do I call a timeout and re-think the play design?
    We are running out of time. We need to make the best decision for our business!

Planning the transition to NFSv4.x

Like any good coach or storage administrator, we need to come into every game prepared. We need to think out all possible scenarios and plan for them accordingly, but also be flexible enough to adjust on the fly if unforeseen circumstances occur. After all, unforeseen circumstances are, by definition, unforeseen.

In TR-4067, I cover some considerations that need to be made when deciding to move from NFSv3 to NFSv4.x. The following is a sneak preview. Keep in mind that this section may change in the final release of the TR, but the general concepts remain.

Transitioning from NFSv3 to NFSv4.x: Considerations

The following section covers some considerations that need to be addressed when migrating from NFSv3 to NFSv4.x. When choosing to use NFSv4.x after using NFSv3, you cannot simply turn it on and have it work as expected. There are specific items to address, such as:

  • Domain strings/ID mapping
  • Storage failover considerations
  • Name services
  • Firewall considerations
  • Export policy rule considerations
  • Client support
  • NFSv4.x features and functionality

ID Domain Mapping

While customers prepare to migrate their existing setup and infrastructure from NFSv3 to NFSv4, some environmental changes must be made before moving to NFSv4. One of them is “id domain mapping.”

In clustered Data ONTAP 8.1, a new option called v4-id-numerics was added. With this option enabled, even if the client does not have access to the name mappings, numeric IDs can be sent in the user name and group name fields and the server accepts them and treats them as representing the same user as would be represented by a v2/v3 UID or GID having the corresponding numeric value.

Essentially, this makes NFSv4.x behave more like NFSv3. This also removes the security enhancement of forcing ID domain resolution for NFSv4.x name strings, so whenever possible, keep this option as the default of disabled. If a name mapping for the user is present, however, the name string will be sent across the wire rather than the UID/GID. The intent of this option is to ensure the server never sends “nobody” as a response to credential queries in NFS requests.

To access this command prior to clustered Data ONTAP 8.3, you must be in diag mode. Commands related to diag mode should be used with caution.

Some production environments have the challenge to build new naming service infrastructures like NIS or LDAP for string-based name mapping to be functional in order to move to NFSv4. With the new “numeric_id” option, setting name services does not become an absolute requirement. The “numeric_id” feature must be supported and enabled on the server as well as on the client. With this option enabled, the user and groups exchange UIDs/GIDs between the client and server just as with NFSv3. However, for this option to be enabled and functional, NetApp recommends having a supported version of the client and the server. For client versions that support numeric IDs with NFSv4, please contact the OS vendor.

Note that -v4-id-numerics should be enabled only if the client supports it.

Storage failover considerations

NFSv4.x uses a completely different locking model than NFSv3. Locking in NFSv4.x is a lease-based model that is integrated into the protocol itself rather than separated out as it is in NFSv3 (NLM). From the ONTAP documentation:

In accordance with RFC 3530, Data ONTAP “defines a single lease period for all state held by an NFS client. If the client does not renew its lease within the defined period, all states associated with the client’s lease may be released by the server.” The client can renew its lease explicitly or implicitly by performing an operation, such as reading a file.

Furthermore, Data ONTAP defines a grace period, which is a period of special processing in which clients attempt to reclaim their locking state during a server recovery.

Term Definition (as per RFC-3530)
Lease The time period in which Data ONTAP irrevocably grants a lock to a client.
Grace period The time period in which clients attempt to reclaim their locking state from Data ONTAP during server recovery.
Lock Refers to both record (byte-range) locks as well as file (share) locks unless specifically stated otherwise.

For more information on locking, see the section in this document on NFSv4.x locking. Because of this new locking methodology, as well as the statefulness of the NFSv4.x protocol, storage failover operates differently as compared to NFSv3. For more information, see the section in this document on storage failover and in clustered Data ONTAP.

Name services

If deciding to use NFSv4.x, it is a best practice to centralize the NFSv4.x users in name services such as LDAP or NIS. This allows all clients and clustered Data ONTAP NFS servers to leverage the same resources and guarantees that all names, UID and GIDs will be consistent across the implementation. For more information on name services, see TR-4073: Secure Unified Authentication and TR-4668: Name Services Best Practices.

Firewall considerations

NFSv3 required several ports to be opened for ancillary protocols such as NLM, NSM, etc. in addition to port 2049. NFSv4.x requires only port 2049. If wishing to use NFSv3 and NFSv4.x in the same environment, all relevant NFS ports should be opened. These ports are referenced in this document.

Volume language considerations

In NetApp’s Data ONTAP, volumes can have specific languages set. This is intended to be used for internationalization of file names for languages that use characters not common to English, such as Japanese, Chinese, German, etc. When using NFSv4.x, RFC-3530 states that UTF-8 is recommended.

11.  Internationalization

   The primary issue in which NFS version 4 needs to deal with
   internationalization, or I18N, is with respect to file names and
   other strings as used within the protocol.  The choice of string
   representation must allow reasonable name/string access to clients
   which use various languages.  The UTF-8 encoding of the UCS as
   defined by [ISO10646] allows for this type of access and follows the
   policy described in "IETF Policy on Character Sets and Languages",
   [RFC2277].

If you intend to migrate to clustered Data ONTAP from a 7-mode system and use NFSv4.x, you should be using some form of UTF-8 language support, such as C.UTF-8 (which is the default language of volumes in clustered Data ONTAP). If the 7-mode system is not already using a UTF-8 language, then it should be converted before transitioning to cDOT or when intending to transition from NFSv3 to NFSv4. The exact UTF-8 language specified will depend on the specific requirements of the native language to ensure proper display of character sets.

7-mode allowed volumes that hosted NFSv4.x data to use C language types. Clustered Data ONTAP does not, as it honors the RFC standard recommendation of UTF-8. TR-4160: Secure Multi-tenancy Considerations covers language recommendations in cDOT. When changing a volume’s language, every file in the volume must be accessed after the change to ensure they all reflect the language change. This can be done via a simple “ls -lR” to do a recursive listing of files.

Export policy rules

In clustered Data ONTAP, it is possible to specify which version of NFS is supported for an exported filesystem. If an environment was configured for NFSv3 and the export policy rule option –protocol was limited to allow NFSv3 only, then it would need to be modified to allow NFSv4. Additionally, policy rules could be configured to allow only NFSv4.x clients access.

Example:

cluster::> export-policy rule modify -policy default -vserver NAS -protocol nfs4For more information, consult the product documentation for your specific version of clustered Data ONTAP.

Client considerations

When using NFSv4.x, clients are as important to consider as the NFS server. The following client considerations should be followed when implementing NFSv4.x. Other considerations may be necessary. Please contact the OS vendor for specific questions about NFSv4.x configuration.

  • x is supported.
  • The fstab file and NFS configuration files are configured properly. When mounting, the client will negotiate the highest NFS version available with the NFS server. If NFSv4.x is not allowed by the client or fstab specifies NFSv3, then NFSv4.x will not be used at mount.
  • The idmapd.conf file is configured with the proper settings.
  • The client either contains identical users and UID/GID (including case sensitivity) or is using the same name service server as the NFS server/clustered Data ONTAP SVM.
  • If using name services on the client, the client is configured properly for name services (nsswitch.conf, ldap.conf, sssd.conf, etc.) and the appropriate services are started, running and configured to start at boot.
  • The NFSv4.x service is started, running and configured to start at boot.

NFSv4.x Features and Functionality

NFSv4.x is the next evolution of the NFS protocol and enhances NFSv3 with new features and functionality, such as referrals, delegations, pNFS, etc. These features are covered throughout this document and should be factored in to any design decisions for NFSv4.x implementations.

If done right and you make the right play calls, NFSv4.x could bring home the championship for  your NAS storage environment!