ONTAP 9.3 NFS sneak preview: Mount and security tracing

aid1871175-v4-728px-trace-step-6-version-2

ONTAP 9.3 is on its way, and with it comes some long-awaited new functionality for NFS debugging, including a way to map volumes to IP addresses!

Mount trace

In ONTAP 7-Mode, you could trace mount requests with an option “nfs.mountd.trace.” That didn’t make its way into ONTAP operating in cluster mode until ONTAP 9.3. I covered a long and convoluted workaround in How to trace NFSv3 mount failures in clustered Data ONTAP.

Now, you can set different levels of debugging for mount traces via the cluster CLI without having to jump through hoops. As a bonus, you can see which data LIF has mounted to which client, to which volume!

To enable it, you would use the following diag level commands:

::*> debug sktrace tracepoint modify -node [node] -module MntTrace -level [0-20] -enabled true

::*> debug sktrace tracepoint modify -node [node] -module MntDebug -level [0-20] -enabled true

When enabled, ONTAP will log the mount trace modules to the sktrace log file, which is located at /mroot/etc/mlog/skrace.log. This file can be accessed via systemshell, or via the SPI interface. Here are a few of the logging levels:

4 – Info
5 – Error
8 – Debug

When you set the trace level to 8, you can see successful mounts, as well as failures. This gives volume info, client IP and data LIF IP. For example, this mount was done from client 10.63.150.161 to data LIF 10.193.67.218 of vserverID 10 on the /FGlocal path:

cluster::*> debug log sktrace show -node node2 -module-level MntTrace_8
Time TSC CPU:INT Module_Level
--------------------- ------------------------ ------- -------------------
 LogMountTrace: Mount access granted for Client=10.63.150.161
 VserverID=10 Lif=10.193.67.218 Path=/FGlocal

With that info, we can run the following command on the cluster to find the SVM and volume:

::*> net int show -address 10.193.67.218 -fields lif
 (network interface show)
vserver lif
------- ---------
DEMO    10g_data1

::*> volume show -junction-path /FGlocal -fields volume
vserver volume
------- ---------------
DEMO    flexgroup_local

The mount trace command can also be used to figure out why mount failures may have occurred from clients. We can also leverage performance information from OnCommand Performance Manager (top clients) and per-client stats to see what volumes might be seeing large performance increases and work our way backward to see what clients are mounting what LIFs, nodes, volumes, etc. with mount trace enabled.

Security trace (sectrace)

In ONTAP 9.2 and prior, you could trace CIFS/SMB permission issues only, using “sectrace” commands. Starting in ONTAP 9.3, you can now use sectrace on SMB and/or NFS. This is useful to troubleshoot why someone might be having access to a file or folder inside of a volume.

With the command, you can filter on:

  • Client IP
  • Path
  • Windows or UNIX name

Currently, sectrace is not supported on FlexGroup volumes, however.

cluster::*> sectrace filter create -vserver DEMO -index 1 -protocols nfs -trace-allow yes -enabled enabled -time-enabled 60

Warning: Security tracing for NFS will not be done for the following FlexGroups because Security tracing for NFS is not supported for FlexGroups: TechONTAP,flexgroupDS,flexgroup_16,flexgroup_local.
Do you want to continue? {y|n}: y

Then, I tested a permission issue.

# mkdir testsec
# chown 1301 testsec/
# chmod 700 testsec
# su user
$ cd /mnt/flexvol/testsec
bash: cd: /mnt/flexvol/testsec: Permission denied

And this was the result:

cluster::*> sectrace trace-result show -vserver DEMO

Node            Index Filter Details             Reason
--------------- ----- -------------------------- ------------------------------
node2           1     Security Style: UNIX       Access is allowed because the
                      permissions                user has UNIX root privileges
                                                 while creating the directory.
                                                 Access is granted for:
                                                 "Append"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is allowed because the
                      permissions                user has UNIX root privileges
                                                 while setting attributes.
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is allowed because the
                                                 permissions user has UNIX root privileges
                                                 while setting attributes.
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is not granted for:
                      permissions                "Modify", "Extend", "Delete"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /
                      Win-User: -
                      UNIX-User: 7041
                      Session-ID: -
node2           1     Security Style: UNIX       Access is not granted for:
                      permissions                "Lookup", "Modify", "Extend",
                                                 "Delete", "Read"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 7041
                      Session-ID: -

As you can see above, the trace output gives a very clear picture about who tried to access the folder, which folder had the error and why the permission issued occurred.

Bonus Round: Block Size Histograms!

Now, this isn’t really a “new in ONTAP 9.3” thing; in fact, I found it as far back as 9.1. I just hadn’t ever noticed it before. But in ONTAP, you can see the block sizes for NFS and CIFS/SMB operations in the CLI with the following command:

cluster::> statistics-v1 protocol-request-size show -node nodename

When you run this, you’ll see the average request size, the total count and a breakdown of what block sizes are being written to the cluster node. This can help you understand your NAS workloads better.

For example, this node runs mostly a VMware datastore workload

cluster::> statistics-v1 protocol-request-size show -node node2 -stat-type nfs3_read

Node: node2
Stat Type: nfs3_read
                     Value    Delta
--------------       -------- ----------
Average Size:        30073    -
Total Request Count: 92633    -
0-511:                1950    -
512-1023:                0    -
1K-2047:              1786    -
2K-4095:              1253    -
4K-8191:             18126    -
8K-16383:              268    -
16K-32767:            4412    -
32K-65535:             343    -
64K-131071:           1560    -
128K - :             62935    -

When you run the command again, you get a delta from the last time you ran it.

If you’re interested in more ONTAP 9.3 feature information, check out Jeff Baxter’s blog here:

https://blog.netapp.com/announcing-netapp-ontap-9-3-the-next-step-in-modernizing-your-data-management/

You can also see me dress up all fancy and break down the new features at a high level here:

I’ll also be doing more detailed blogs on new features as we get closer to the release.

Advertisements

5 thoughts on “ONTAP 9.3 NFS sneak preview: Mount and security tracing

  1. When I try to mount a qtree, I get an access denied message on my Linux client and on the filer I see:

    Node Index Filter Details Reason
    ————— —– ————————– ——————————
    filer-01 1 Security Style: NTFS and Access is not granted for:
    NT ACL “Lookup”, “Modify”, “Extend”,
    “Delete”, “Read”
    Protocol: nfs
    Volume: filer_vol01
    Share: –
    Path: /
    Win-User: –
    UNIX-User: 0
    Session-ID: –

    I used this command on Linux:
    sudo mount -v -o sec=sys,vers=4 svm1:/vol/vol1/projects /mnt
    I can mount /vol/vol1, but not the full path. Do you have any idea why and how it can be fixed?

    Like

    • Not sure if you figured this out yet or not, but you should be able to mount subdirs. I can do it in my lab:

      [root@centos7 /]# mount -o sec=sys,nfsvers=4 demo:/flexvol/tree/dir /flexvol
      [root@centos7 /]# mount | grep flexvol
      demo:/flexvol/tree/dir on /flexvol type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.193.67.225,local_lock=none)

      The “NTACL” message suggests the security style of the object is NTFS, so that means the UNIX user should be able to map to a valid Windows user to allow it to mount.

      You can check the security style of the object with:

      ontap9-tme-8040::> vserver security file-directory show -vserver DEMO -path /flexvol/tree/dir

      Vserver: DEMO
      File Path: /flexvol/tree/dir
      File Inode Number: 1037808
      Security Style: unix
      Effective Style: unix
      DOS Attributes: 10
      DOS Attributes in Text: —-D—
      Expanded Dos Attributes: –
      UNIX User Id: 0
      UNIX Group Id: 0
      UNIX Mode Bits: 755
      UNIX Mode Bits in Text: rwxr-xr-x
      ACLs: –

      And you can check the user name mapping with:

      ::> set diag
      ::*> diag secd authentication show-creds -node ontap9-tme-8040-02 -vserver DEMO -unix-user-name root

      UNIX UID: root Windows User: DEMO\Administrator (Windows Local User)

      GID: daemon
      Supplementary GIDs:
      daemon

      Primary Group SID: DEMO\None (Windows Domain group)

      Windows Membership:
      BUILTIN\Administrators (Windows Alias)
      User is also a member of Everyone, Authenticated Users, and Network Users

      Privileges (0x2237):
      SeBackupPrivilege
      SeRestorePrivilege
      SeTakeOwnershipPrivilege
      SeSecurityPrivilege
      SeChangeNotifyPrivilege

      Like

  2. Your reply pointed me in the right direction and I got a bit further, but I have not fully understood how to resolve the problem yet.

    When I change the default Windows user to my NT user, I can mount the share. But then all unknown users have the same access rights as I have. So I changed the default Windows user to pcuser, which only exist as a local user on the filer and added a name mapping for unix user root to my NT user.

    But that again gives me a access denied/permission denied error when trying to mount the qtree. So I changed the export policy to allow super user access and the mount worked, but root should not have permissions to access files and directories. I only want to allow root to mount the qtree.

    Seems like I am messing things up here, but I have no clue on where I am wrong. May be you can shed some light on this?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s