Mixed perceptions with NetApp multiprotocol NAS access

EDIT: As the original post for this was super long, I’ve since broken it up into a 2 part post. I moved the vserver security information to the following post:

Managing ACLs via the ONTAP Command Line

EDIT #2 April 8, 2021: I FINALLY WROTE THE MULTIPROTOCOL TR! 

Brand new tech report: Multiprotocol NAS Best Practices in ONTAP

NetApp’s ONTAP operating system is one of the few storage operating systems out there that supports data access from both CIFS/SMB and NFS clients. NetApp’s been doing this for a long time – longer than I’ve been there, and I’m going on 10 years!

Despite the fact that it’s been around so long and is one of *the* core competencies in ONTAP, it’s one of the most frequently misunderstood configurations I see. When I was in support, it was one of the biggest case generators. As the NFS TME, it’s one of the most common emails I get that customers need assistance on.

I can tell you what it’s not….

Multiprotocol NAS is NOT “Mixed Mode”

Many people use this terminology for describing access from multiple clients. Unfortunately, it only adds to the confusion, because there is also a security style called “mixed” (see below) and that makes people associate the two and then they start setting mixed security styles when they don’t need to…

So, call it what it is – Multiprotocol NAS. 🙂

What’s so hard about it?

The reason it seems to confound so many people is two fold:

  • Windows administrators are generally not UNIX-savvy
  • UNIX administrators are generally not Windows-savvy

To truly understand multiprotocol NAS, you either have to know both Windows and UNIX file systems/security sematics pretty well, or be open to the fact that Windows and UNIX have similarities and differences.

That said, when you do understand how it works and get it configured properly, it’s a pretty powerful tool for serving data for multiple client types.

There’s currently a Multiprotocol TR in the works, but will be a ways out. However, I just dealt with a recent multiprotocol NAS issue and wanted to do a brain dump before the information got stale and I had to revisit it. This blog is intended to be a quick hit guide to multiprotocol NAS in ONTAP. Some of the ideas will make their way into official TR format.

What makes multiprotocol NAS possible in ONTAP?

ONTAP is fairly agnostic when it comes to file systems and ACL styles. SMB and NFS clients use different security semantics, but the general concepts of those are the same.

Users, groups, permissions.

From there, things tend to skew a bit. Windows uses NTFS security concepts. NFS clients use mode bits for NFSv3/NFSv4.x or ACLs for NFSv4.x. NFSv3 had the concept of POSIX ACLs, but ONTAP doesn’t support those.

The issue is that NTFS ACLs are more complex than mode bits, but match up pretty nicely with NFSv4.x ACLs. Mode bits only do Read, Write, eXecute (RWX), so Windows ACLs don’t match up 1 to 1, especially when you have “special permissions” in the mix. As a result, when dealing with ONTAP file systems, we have the concept of a security style that helps us choose the style of ACL we want to implement. The choices we have:

  • NTFS – NTFS ACLs only
  • UNIX – UNIX style permissions only
  • Mixed – UNIX or NTFS permissions, depending on who last changed permissions
  • Unified (Infinite Volume only)

To properly address permissions, ONTAP has to pick one security style over the other. This allows the storage system to decide which direction a user will map to determine the correct permissions. After all, what’s the point of permissions if they don’t work properly?

User mapping

ONTAP is not unique in the concept of user mapping, but it is still a concept that gets people confused on occasion.

Essentially, to get the proper permissions on a NetApp storage system, a client must first pass a “test” in the form of initial authentication.

The initial test is “Who are you?”

The storage system needs to know that the user you are claiming to be is actually you. There are varying degrees of how secure this test is, mostly dependent on the protocol you’re using, but the bottom line is this: authentication helps us get a user name. That user name allows us to map to another user name, depending on the volume security style.

In general:

  • SMB clients always map to a UNIX user because ONTAP is UNIX-based, even if NTFS security style is in use
  • If no name mapping rules or 1:1 name mappings exist, SMB users map to a default UNIX user set in CIFS options (pcuser/65534 by default)
  • 65534 is “nobody” or “nfsnobody” in most UNIX clients
  • NFS clients only map to Windows users when the security style is NTFS
  • NFS clients cannot chmod or chown on NTFS style volumes; SMB clients cannot take ownership or change ACLs on UNIX style volumes

Once a user has authenticated, the permissions can be discerned based on access control lists. One can see those ACLs via the CLI of the storage system with “vserver security file-directory show.”

cluster::*> vserver security file-directory show -vserver parisi -path /cifs

                 Vserver: parisi
               File Path: /cifs
       File Inode Number: 64
               Security Style: ntfs
         Effective Style: ntfs
          DOS Attributes: 10
  DOS Attributes in Text: ----D---
 Expanded Dos Attributes: -
            UNIX User Id: 0
           UNIX Group Id: 0
          UNIX Mode Bits: 777
  UNIX Mode Bits in Text: rwxrwxrwx
                    ACLs: NTFS Security Descriptor
                          Control:0x8004
                            Owner:BUILTIN\Administrators
                            Group:BUILTIN\Administrators
                            DACL - ACEs
                             ALLOW-Everyone-0x1f01ff
                             ALLOW-Everyone-0x10000000-OI|CI|IO

User/name mapping is one of the most important pieces of the multiprotocol NAS puzzle. Get that part right and most everything else is easy.

Name mapping can be done either locally (via name mapping rules) or with LDAP. TR-4073 covers this sort of thing in pretty finite detail.

Name services/LDAP

The easiest way to handle name mapping in ONTAP for multiprotocol NAS is to leverage a name service server like LDAP. When dealing with both SMB and NFS, the most logical choice is to use the existing Active Directory infrastructure to host UNIX identities. While you can host name mapping rules for users that don’t have the same UNIX and Windows names, it’s best to try to have UNIX and Windows user names match 1:1. (I.e., DOMAIN\nfsdudeabides == nfsdudeabides in UNIX).

TR-4073 covers LDAP and TR-4379 covers name service best practices for ONTAP 9.2 and prior. TR-4668 covers name services in ONTAP 9.3 and beyond.

Mixed Security Style

Fun fact – Mixed security style isn’t truly “mixed.” When you use mixed security style, it’s always either NTFS or UNIX security style at any given moment. This is known as the “effective” security style, which can be seen in “vserver security file-directory show.”

cluster::*> vserver security file-directory show -vserver parisi -path /cifs

                 Vserver: parisi
               File Path: /cifs
       File Inode Number: 64
               Security Style: ntfs
         Effective Style: ntfs

The “effective” style changes based on  the last permission change. If an NFS client does a chmod or chown, the mixed security style volume changes to effective UNIX security style. If an SMB client changes owner or sets an ACL, the effective security style changes to NTFS. When these effective styles change, how the storage does name mapping changes (ie; win-unix to unix-win, etc).

Is mixed security style recommended?

Generally speaking, you don’t want file systems changing something behind the scenes without the knowledge of the storage administrators. Plus, these changes can affect functionality, and even access. As a result, mixed security style is generally not recommended. The only time you’d want to use mixed security style is if your environment requires the ability for clients or applications to change permissions from both NFS and SMB. And even then, if you do set up mixed security style, consider limiting the ability for regular users to take ownership or change permissions on folders and files via NTFS ACLs.

Otherwise, I personally recommend picking either NTFS or UNIX and sticking with it. That choice would be based on how you want your users to manage their ACLs, as well as how granular you want control to be on those file systems. For example, mode bits in UNIX only allow setting an owner, group and everyone else. There’s no way to set multiple groups with different access on the object unless you use NFSv4 or NTFS ACLs.

I usually prefer NTFS because you get the granularity, as well as the GUI functionality many users are accustomed to.

If you do decide to use mixed security style, keep the following in mind:

  • If a volume is using mixed security style and the effective style gets flipped from NTFS to UNIX and then back to NTFS by way of the clients, the previous NTFS ACLs are lost.
  • When a volume flips from UNIX effective to NTFS effective, you get the mode bit translation. For example, if the UNIX volume was 755, you get “Owner – Full Control” and “Everyone – Read/Execute” as Windows ACLs. 700 gives “Owner – Full Control” only.
  • Administrator always gets added onto the ACL with Read/Write access when we flip to NTFS from UNIX.
  • With mixed security style, there are two types of owners – UNIX owner and Windows owner. When Windows “takes ownership,” the UNIX owner does not change.
  • When the effective style of the volume is NTFS, UNIX clients will see permissions as 777 unless the NFS server option ntacl-display-permissive-perms is set to “disabled.”

For information on how to manage permissions in ONTAP, see the following post:

Managing ACLs via the ONTAP Command Line

Be on the lookout for a multiprotocol TR in the future that covers this and more!

Got any questions? Feel free to post in the comments!

37 thoughts on “Mixed perceptions with NetApp multiprotocol NAS access

  1. Pingback: Managing ACLs via the ONTAP Command Line | Why Is The Internet Broken?

  2. Hi Justin,
    Many thanks for your article, very interesting !
    Just one question,

    We have a netapp volume (NTFS mode), many users and groups AD have different rights in this share, no problem,

    for some project we will need to create or delete some files in this volume from a linux account (Redhat server), regarding exportfs, usermap.cfg, everything seems to be ok, but effectively i see data in this volume with the root account but not with other linux account,

    usermap.cfg

    domain_name\linuxaccount123==root
    OK

    domain_name\linuxaccount123==linuxaccount123
    NOK

    I created the account linuxaccount123 in our AD, this account has R/W rights in the cifs share, but unable to see data from the linux mount –> permission denied

    Is there any other things to do ?

    thanks for your help

    Best regards

    Speedypizz

    Like

  3. Justin –
    I have what may be a related question. We migrated a ton of data from an old Solaris system running ZFS with (ACLs) to NFSv4 running on a NetApp filer which we have mapped to a Linux cluster. We wrote a conversion script to pick up the ZFS ACLs and put them down as NFSv4. The Linux servers accessing the NFSv4 all use Active Directory authentication and all the id’s and ownership map through properly to the NFSv4 volume. We run Samba on one of the Linux servers and users can access anything where the Linux file mask lets them in, but of course the NFSv4 ACLs are not honored (they got around this on Solaris as they had a special vfs module which supported ZFS ACLs in Samba), this doesn’t appear to be an option for Samba and NFSv4. Long story short, I am wondering if we can put the NetApp filer in a mode so that users could access the volume directly from Windows as a CIFS share and respect the ACLs present on the NFSv4 share.

    Like

    • If the users map properly and you use UNIX security style, you should be able to access volumes from Windows CIFS and honor the existing v4 ACLs. However, if you’re planning on going to Windows anyway, it might be better in the long run to use NTFS security style so that Windows users can change ACLs. UNIX security style won’t allow permission changes from Windows.

      That said, converting v4 ACLs to NTFS ACLs will be a pain in the ass, so it may be easier to simply leave as is if you don’t need Windows users to be able to modify ACLs.

      Like

  4. Hi Justin,
    thanks for clarification on this. I was one of the users who got confused by “mixed mode” definition and not aware of username-mapping.

    Like

  5. Pingback: NetApp Multiprotocol Tutorial Part 1 - FlackBox

  6. Pingback: NetApp Multiprotocol Tutorial Part 2 - FlackBox

  7. Justin,
    Great explanation on multi-protocol use. We have been using it for the past 10 years successfully since becoming NetApp customers. But, we recently went from 7-Mode to Cluster-Mode (9.1) and now have an issue using CIFS to access our volumes that all use Unix Security Style. We utilize LDAP and AD with an add-on from Beyond Trust so that we can add GIDs and UIDs to our groups and users. All our users are 1:1 in AD. In the past under 7-Mode, when we added a user to a group or removed a user from a group the user would simply log off and on and the change would be in effect. But now under Cluster-Mode, after we make a change to a group it takes hours and hours (24 hrs. in some cases) for the NetApp to recognize the change to the group that was made even if the user logs off and on multiple times. I have tried many things, including clearing some of the ONTAP caches, etc. but with no luck. The change is propagated through the network to all domain controllers within a minute or two. The issue does not affect NFS.

    I currently, have a case open with NetApp support, but we haven’t received a resolution yet. Any help you can provide would be appreciated. Since you are with NetApp, if you would like the case number you can email me and I will provide it.

    Like

    • This is definitely a cache of some sort. ONTAP 9.3 and later does a better job with cache consolidation and management. TR-4379 covers caches for ONTAP 8.3 up until around 9.1. TR-4678 is an update to that and covers caches in ONTAP 9.3.

      24 hour cache sounds like the secd cache or an nblade cache. One surefire way to verify that is to have the user connect to a different node via a different IP address, where they have not yet populated the cache.

      Like

  8. Pingback: How to find average file size and largest file size using XCP | Why Is The Internet Broken?

  9. Hi Justin, thanks for the great post. We are struggling with sharing data in a mixed Unix/Windows environment and what you describe looks like an very interesting and attractive solution. My question is if there are known drawbacks of multiprotocol NAS (e.g. data security and permissions / user identities) ? Is it difficult / time consuming to configure and maintain? Or is it straight forward to set up?

    Like

    • The main drawbacks of multiprotocol NAS are:

      – Permissions won’t map 1:1 with NFSv3/NTFS. But, NFSv4.x ACLs do.
      – How users are able to view ACLs will vary depending on configuration
      – How users set ACLs will vary depending on configuration

      Data security shouldn’t be an issue, unless you don’t set permissions correctly. User identities are a challenge in that you need to map users, but if you use LDAP and keep usernames the same across NFS and SMB, should be a piece of cake. Once it’s configured, it will work fine.

      If you want a more in depth discussion, reach out to your NetApp sales rep and we can start a discussion.

      Like

  10. Hi Justin,

    I just witnessed a discussion at work where it was stated: we cant backup mixed security as restoring would be a risk.

    I am thinking, nah.. must be possible?

    Surely (?) its possible to have a dedicated SVM for restore purposes from mixed security env. SVM could be configured to “bypass” all of the drama/risks associated with this? Something like, enable both protocols on SVM, use a local user that allows for any file to be restored on the volume/qtree and then expose (NFS or SMB) that out as needed?

    If I am wrong, happy to be called out 😉
    Cheers,
    Eric

    Like

    • I’m curious as to what the risk of restoring would be. Backing up mixed security simply backs up at the file level. Whatever effective security style is present on a file or folder at the time of backup is what you get on restore. You don’t even need a dedicated SVM; files can be pulled from backup or from snapshot.

      Like

  11. Thanks for clarification, Justin.

    In fact, we have a pretty simple Windows-user = Linux-user setup. You wrote that the way users set/view permissions will depend on configuration (and that e.g. UNIX clients cannot chmod/chown on NTFS security style volums). That should be OK for users, but does that significantly affect the way the storage admins will handle permissions on the filer?

    Actually, my company is already a NetApp customer. However, cause we were a “Windows-only-shop” for quite some time, we have had only CIFS shares. Changing running systems to better integrate Linux/UNIX that are coming in is of course not always very popular.

    In a recent meeting, when I proposed the idea of looking into Multiprotocol on our NetApp filer(s), our storage admins claimed that multi-protocol is “only working on paper and in marketing material, but not usable in real world production” and that is is “an administrative and dependency hell”. They also stated that not even the local technical support from NetApp was able to configure it in a way that works properly in our company in a test they conducted. So, they were unfortunately pretty categorical excluding it. In consequence I am stuck with cumbersome workarounds, so I really would like to get that working somehow…

    I tend to think this is a “defensive maneuver” and I really have a hard time to buy into those claims, esp. given the long time multi-protocol has been on the market (and reading you naming it *the* core functionality) . Unfortunately, I have no way to challenge the claims I am confronted with myself (no access to the storage infrastructure), so my question is, are you aware of any evaluation or reference implementation, I could use as an argument?

    Like

    • We have a sizable number of customers using SMB and NFS on the same datasets in production with no issues. The “administrative hell” is generally associated with lack of large scale infrastructure to support both SMB and NFS users from a central location (such as LDAP) or an environment that incorporates usernames with different Windows and UNIX names. What your colleagues may be referring to is “mixed security style,” which indeed is not recommended unless you have a specific need to be able to modify permissions from NFS and SMB clients on the same files, and is likely what NetApp support advised against.

      I’m not sure what “local technical support” they’re referring to, but I’ve done a number of live demos of this working, and there is a Hands on Lab that you could gain access to via your sales rep.

      Your sales rep could also probably set up a call with your team and myself, as well as find references using SMB/NFS in the same environments.

      Like

      • Thanks Justin, for your reply. I will forward this information to my colleagues and post again what progress we made on this issue…

        Like

      • Hi Justin,

        and many thanks again for your kind offer and support.

        Just want to report back, that my colleagues unfortunately do not seem to be willing to investigate possibilities with Multi-Protocol any further (which I do not understand at all). That makes me conclude, that if there are any challenges with Multi-Protocol on NetApp filers, that these are rather on our side (in particular 50 cm in front of some of the screens here)…

        Again, thanks for your kind and patient replies, very much appreciated!

        Cheers
        Stef

        Like

  12. Pingback: Windows NFS? WHO DOES THAT??? | Why Is The Internet Broken?

  13. Justin, Thanks for article and the reference TRs. It was very insightful. I have a query on this.

    In a multi protocol file access environment where Linux OS users are accessing files from NFS mount which has NTFS security style, if the namemap database is set to ‘files’ only (not including ‘ldap’) and we have a corresponding name mapping (vserver name-mapping show) also configured for example (Direction: unix-win, Position: 1, Pattern: *, Replacement: *\\*) what is the purpose of reaching LDAP server or using cached credentials of LDAP server? Please refer to following name mapping example for reference.

    **********************************************************************************************************************
    [ 13] Determined UNIX id 123456789 is UNIX user ‘myuser’
    [ 13] Mapping Successful for Unix-user ‘myuser’ to Windows user ‘*\myuser’ at position 1
    [ 13] Wildcard domain mapping
    [ 57] Using a cached connection to mydc01.na.mydomain.com
    [ 99] Could not find Windows name ‘ROOT\myuser’


    [ 444] Could not find Windows name ‘na\myuser’
    [ 444] Unable to find the user ‘*\myuser’ in any trusted domains.
    **[ 445] FAILURE: Name mapping for UNIX user ‘myuser’ failed. Explicit Mapping failed and no default mapping found
    **********************************************************************************************************************

    Like

    • An LDAP server would provide a centrally located, centrally managed resource for both clients and ONTAP SVM alike. They would all reference the same database of users, groups and name mappings, rather than having to keep local files around.

      In the case of your example, the Windows user “myuser” doesn’t exist in your NA domain, so the mapping from UNIX to Windows fails. It’s attempting to map to a Windows user because it needs one to resolve the proper NTFS credentials. In your example, it looks like you have a local UNIX user defined. If that user name was the same as a Windows user or if you had an explicit name mapping rule defined, that would map the user properly.

      Like

      • Many Thanks for your quick reply! Yes that unix user has corresponding domain account and name mapping happens as expected. But we troubleshooted once the user failed to get authenticated and found that during that time of failure that LDAP server in my example was down (in my example: mydc01.na.mydomain.com). So after that i was trying to find out why reached that LDAP server not any other working LDAP server or LDAP servers which are listed in ldap client show (note that DC is not listed anywhere in filer).

        So i thing i understood from your article and TRs that in namemap database if ‘ldap’ is added as source, then it will consult DC that are listed in ldap client configured in the filer (ldap client show). And also the requirement is symmetric mapping not asymmetric i.e unix user ‘myuser’ maps to domain user ‘na\myuser’.

        But what i’m not able to understand, in case of the namemap database source is set to ‘files’ only, then how filer decides which DC to connect? which file or configuration dictates filers to connect to this particular LDAP server (like in my example mydc01.na.mydomain.com which is not in ldap client list)? And if that LDAP server is not responding does it tries to reach any other LDAP server?

        Like

      • Namemap is only relevant if you need to do asymmetric name mapping (for example user “john” in UNIX maps to user “jack” in Windows). ONTAP will attempt to map a username 1:1 from UNIX Windows without needing a name mapping. So if the names all match, there’s no need to specify namemap. If namemap is set to files, it never tries to connect. It tries 1:1 and then tries the local user mappings specified on the SVM.

        If LDAP is not responding, it will try other LDAP servers in the list.

        Like

      • One thing i noticed yesterday while going through this article the link for TR-4379 is opening TR-4073. The link for TR-4073 is good. I observed this in below line.

        “TR-4073 covers LDAP and TR-4379 covers name service best practices, for your reference.”

        I just figured that out by replacing the link in browser by the TR name (https://www.netapp.com/us/media/tr-4379.pdf) which all other readers have done i presume.

        As this article is so relevant and informative for readers looking to understand multiprotocol file access, i would request you to please update that link so readers dont miss the reference TR-4379 .

        Like

  14. Hello,
    I’ve been trying to get name-mapping working with a multiprotocol share that has Unix security.
    The Unix users use RedHat linux servers and they are authenticated with the SSSD service on each linux server talking to the AD domain controllers. So we are not using any extended schema in AD and we are not using RedHat Directory Services.

    So, after reading thru your various TRs, am I correct in assuming that I would need to enable Kerberos for NFS *or* setup local unix users on the NetApp’s SVM with the uid # generated by the SSSD service for 1:1 name mapping? We are still using NFSv3 … it wasn’t clear what issues I may run into using Kerberos with NFSv3.

    Like

    • You don’t need Kerberos to get name mapping working. Kerberos is simply to encrypt authentication and potentially in-flight data (such as with krb5p).

      SSSD’s configuration will matter here. Is it pulling UIDs from LDAP? Or is it creating UIDs based on the Windows SID? If it’s the latter, the UIDs will be fairly long numerics and uses an algorithm to create them. ONTAP doesn’t support that algorithm to create UIDs from SIDs, so you’d have to create local users in that case, or populate Windows AD with the UNIX attributes/UIDs so that SSSD and ONTAP can pull from the same sources. From the sounds of your config (not extending AD schema), it appears SSSD is doing the algorithm based UID method described here: https://www.samba.org/samba/docs/current/man-html/idmap_rid.8.html

      It all depends on how you want to manage it; do you want to create local users and groups and manage that per SVM or do you want to create a central location that you only have to manage once by populating UNIX attributes in AD?

      Liked by 1 person

  15. SSSD is definitely creating UIDs based on the Windows SID.

    I would definitely prefer to have a central location to manage but that will involve much “paperwork” to get approved (I work in fairly large shop). Since I currently only have one SVM at the moment that needs this requirement, I will have to create the local users and groups.

    Thanks for you help!

    Like

  16. Hello Justin,

    We have configured multi protocol access with name-mapping, but whenever we write from windows, the user is still getting set as nobody. Is there anything you can suggest?

    Like

  17. Pingback: Brand new tech report: Multiprotocol NAS Best Practices in ONTAP | Why Is The Internet Broken?

Leave a comment