Every now and then, the question of mounting CIFS/SMB shares in ONTAP using Linux clients comes up. Many people might consider this configuration to be a crime against nature.
If you’re asking yourself “why would someone want to do this,” there are a few reasons.
- Lock handling – SMB locks use oplocks. NFS locks use shared locks or delegations, depending on the NFS version and options being used. Generally this is fine, but in some cases, the style of lock might create complications with some applications. For example, Kafka on NFS has some issues with how NFS handles deletions when a file is locked:
- Legacy copiers/scanners – Many companies will use copiers/scanners to create image files that redirect to a NAS share in their environment. Many of those copiers use old SMB clients and are no longer under support, so they can’t be upgrades. And many of those scanners/copiers are expensive, so people aren’t interested in buying newer ones just to get newer SMB versions.
- “How we’ve always done it” – This is a common reason that doesn’t always have technical logic associated with it, but you still have to roll with it.
There are other reasons out there as well, I’m sure. But this post will attempt to group the considerations, support statements and issues you might see in case someone else has that question.
Mounting SMB shares in ONTAP – General Guidelines
In 7-Mode ONTAP, mounting SMB shares with clients like AIX had no issues. This was because 7-Mode supported a lot of old SMB technology concepts that newer versions of ONTAP have deprecated. Even though SMB shares on Linux worked in 7-Mode, there was never an official supported configuration for it.
NetApp ONTAP has traditionally only supported Apple OS and Windows OS for CIFS/SMB workloads.
You can find the supported clients in the Interoperability Matrix here:
With newer versions of ONTAP, you *can* get SMB/CIFS working properly, provided you take the following into account:
- CIFS/SMB clients should support UNICODE for CIFS/SMB communication. Clustered ONTAP doesn’t support non-UNICODE in any release and never will. Newer CIFS/SMB clients should support UNICODE. AIX is a good example of an older Linux client that has some challenges using SMB because it still uses non-UNICODE in some client installs.
- CIFS/SMB versions mounting should reflect the supported SMB versions in ONTAP. ONTAP has deprecated SMB1, so you’d need to mount using SMB2 or SMB3.
- CIFS/SMB enabled features in ONTAP should reflect the feature support for the CIFS/SMB Linux clients. For example, my Centos7 Linux Samba client doesn’t support large MTU, SMB3 encryption, multichannel, etc. That can cause failures in the mounts (for an example, see below).
- If using Kerberos for the CIFS/SMB mounts, be sure the CIFS/SMB ONTAP server has a valid SPN for the CIFS service. The name should match the mount name. For example, if your CIFS/SMB server is named “CIFS” then the SPN would be cifs/cifs.domain.com.
Common Issues when Mounting CIFS/SMB
One of the main issues people run into with SMB mounts in Linux (provided they follow the guidelines above when using ONTAP) is using the wrong command syntax. For example, when you specify the share path, you use either \\\\name\\share (extra slashes to tell Linux that the \ isn’t an operand) or you use //name/share.
The following NetApp KB article does a good job of giving command examples for the right way to mount SMB:
Usually the command will tell you if the wrong option is being used, but sometimes the errors are less than helpful. These are a few errors I ran into.
Incorrect Mount Option
This was pretty self-explanatory. First, I didn’t have the cifs-utils installed. Second, I used the wrong command syntax.
Mount point does not exist
Again, self explanatory. The directory you’re trying to mount to needs to exist.
No such file or directory (when mounting)
This means you specified the wrong export path on the NFS server. Check your junction-paths and try again.
Host is down
This one was a tricky error, as it suggests that the server was not up. In some cases, that may truly be the issue. But it wasn’t in my case.
# mount -t cifs -o user=cifsuser \\\\DEMO\\nas /mnt/nas Password for cifsuser@\DEMO\nas: ********** mount error(112): Host is down
But, in reality, the issue was that I didn’t specify the SMB version and it tried to use SMBv1.0 by default.
Specifying the SMB version (-o vers=3.0) got past that issue.
Required key not available
This is a Kerberos specific error. In my case, the cifs/servername.domain.com SPN didn’t exist for the hostname I used in the UNC path. You can see that in a packet capture.
This error is fairly useless in both Windows and Linux in a lot of cases – mainly because it can mean a variety of things. Sometimes, it really is an access issue (such as share or file-level permissions). But in my testing, I also ran into this issue when I had unsupported SMB features enabled on my CIFS server in ONTAP.
# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas Password for administrator@//DEMO/nas: ********** mount error(13): Permission denied
In a packet trace, I could see the client telling me what it supported:
But the reply didn’t really mention the issue. So I made sure to disable the following CIFS/SMB features:
- SMB3 encryption (cifs security modify)
- Large MTU and SMB Multichannel (cifs options modify)
Once I did that, I was able to mount.
[root@centos7 /]# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas Password for administrator@//DEMO/nas: ********** [root@centos7 /]# touch /mnt/nas/smbfile [root@centos7 /]# ls -la /mnt/nas total 1 drwxr-xr-x 2 root root 0 Mar 27 14:36 . drwxr-xr-x. 9 root root 97 Mar 27 10:17 .. -rwxr-xr-x 1 root root 0 Mar 27 14:36 smbfile
For Kerberos, you would just specify sec=krb5.
# mount -t cifs -o sec=krb5,vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas
Stale file handle errors when crossing junction paths
In some cases, if you mount volumes to volumes (via junction-path), newer Linux clients may report different inode numbers for the junctioned volume when using a command like “ls -i” than when using “ls -id”. This can create potential access problems from clients, such as “stale file handle” errors.
For example, this Centos 8.3 client shows different inode numbers for the same volume:
# ls -i 8107029540347314273 mountedvolume # ls -id /mnt/cifs/mountedvolume/ 2744251190362505280 /mnt/cifs/mountedvolume/
The way around this issue is to mount with the mount option “actimeo=0.” Newer SMB clients seem to cache inode information and create potential conflicts. Disabling actimeo prevents that caching from taking place. Since this is a caching option, you might see slower overall performance, but it will fix the stale file handle issue. Alternately, you may want to mount with the “noserverino” option instead, which doesn’t try to use the SMB server’s inodes.
Inode Numbers When Unix Extensions are enabled, we use the actual inode number provided by the server in response to the POSIX calls as an inode number. When Unix Extensions are disabled and "serverino" mount option is enabled there is no way to get the server inode number. The client typically maps the server-assigned "UniqueID" onto an inode number. Note that the UniqueID is a different value from the server inode number. The UniqueID value is unique over the scope of the entire server and is often greater than 2 power 32. This value often makes programs that are not compiled with LFS (Large File Support), to trigger a glibc EOVERFLOW error as this won't fit in the target structure field. It is strongly recommended to compile your programs with LFS support (i.e. with -D_FILE_OFFSET_BITS=64) to prevent this problem. You can also use "noserverino" mount option to generate inode numbers smaller than 2 power 32 on the client. But you may not be able to detect hardlinks properly.
Once I set actimeo=0, this is how those inodes looked:
# ls -i 8107029540347314273 mountedvolume # ls -id /mnt/cifs/mountedvolume/ 8107029540347314273 /mnt/cifs/mountedvolume/
There you go! Hopefully this helped someone from going down a Google K-hole. Leave comments or suggestions below.