ONTAP allows you to mount volumes to other volumes in a Storage Virtual Machine, which provides a way for storage administrators to create their own folder structures across multiple nodes in a cluster. This is useful when you want to ensure the workload gets spread across nodes, but you can’t use FlexGroup volumes for whatever reason.
This graphic shows how that can work:
In NAS environments, a client will ask for a file or folder location and ONTAP will re-direct the traffic to wherever that object lives. This is supposed to be transparent to the client, provided they follow standard NAS deployment steps.
However, not all NAS clients are created equal. Sometimes, Linux serves up SMB and will do things differently than Microsoft does. Windows also will do NFS, but it doesn’t entirely follow the NFS RFCs. So, occasionally, ONTAP doesn’t expect how a client handles something and stuff breaks.
If you’ve ever used a Mac, you’ll know that the Finder can do somethings a little differently than the Terminal does. In this particular issue, we’ll focus on how Finder unzips files (when you double-click the file) in volumes that are mounted to other volumes in ONTAP.
One of our customers hit this issue, and after poking around a little bit, I figured out how to workaround the issue.
Here’s what they were doing:
- SMB to Mac clients
- Shares at the parent FlexVol level (ie, /vol1
- FlexVols mounted to other FlexVols several levels deep (ie, /vol1/vol2/vol3)
When files are unzipped after accessing a share at a higher level and then drilling down into other folders (which are actually FlexVols mounted to other FlexVols), then unzipping in Finder via double-click fails.
When the shares are mounted at the same level as the FlexVol where the unzip is attempted, unzip works. When the Terminal is used to unzip, it works.
However, when your users refuse to use/are unable to use the Terminal and you don’t want to create hundreds of shares just to work around one issue, it’s an untenable situation.
So, I decided to dig into the issue…
Reproducing the issue
The best way to troubleshoot problems is to set up a lab environment and try to recreate the problem. This allows you freedom to gather logs, packet traces, etc. without bothering your customer or end user. So, I brought my trusty old 2011 MacBook running OS Sierra and mounted the SMB share in question.
These are the volumes and their junction paths:
DEMO inodes /shared/inodes DEMO shared /shared
This is the share:
Vserver: DEMO Share: shared CIFS Server NetBIOS Name: DEMO Path: /shared Share Properties: oplocks browsable changenotify show-previous-versions Symlink Properties: symlinks File Mode Creation Mask: - Directory Mode Creation Mask: - Share Comment: - Share ACL: Everyone / Full Control File Attribute Cache Lifetime: - Volume Name: shared Offline Files: manual Vscan File-Operations Profile: standard Maximum Tree Connections on Share: 4294967295 UNIX Group for File Create: -
I turned up debug logging on the cluster (engage NetApp Support if you want to do this), got a packet trace on the Mac and reproduced the issue right away. Lucky me!
I also tried a 3rd party unzip utility (Stuffit Expander) and it unzipped fine. So this was definitely a Finder/ONTAP/NAS interaction problem, which allowed me to focus on that.
Packet traces showed that the Finder was attempting to look for a folder called “.TemporaryItems/folders.501/Cleanup At Startup” but couldn’t find it – and couldn’t create it, apparently either. But it would created folders named “BAH.XXXX” instead, and they wouldn’t get cleaned up.
So, I thought, why not manually create the folder path, since it wasn’t able to do it on its own?
You can do this through Terminal, or via Finder. Keep in mind that the path above has “folders.501” – 501 is my uid, so check your users uid on the Mac and make sure the folder path is created using that uid. If you have multiple users that access the share, you may need to create multiple folders.xxx in .TemporaryItems.
If you do it via Finder, you may want to enable hidden files. I learned how to do that via this article:
So I did that and then I unmounted the share and re-mounted, to make sure there wasn’t any weird cache issue lingering. You can check CIFS/SMB sessions, versions, etc with the following command, if you want to make sure they are closed:
cluster::*> cifs session show -vserver DEMO -instance Vserver: DEMO Node: node1 Session ID: 16043510722553971076 Connection ID: 390771549 Incoming Data LIF IP Address: 10.x.x.x Workstation IP Address: 10.x.x.x Authentication Mechanism: NTLMv2 User Authenticated as: domain-user Windows User: NTAP\prof1 UNIX User: prof1 Open Shares: 1 Open Files: 1 Open Other: 0 Connected Time: 7m 49s Idle Time: 6m 2s Protocol Version: SMB3 Continuously Available: No Is Session Signed: true NetBIOS Name: - SMB Encryption Status: unencrypted Connection Count: 1
Once I reconnected with the newly created folder path, double-click unzip worked perfectly!
Check it out yourself:
Note: You *may* have to enable the option is-use-junctions-as-reparse-points-enabled on your CIFS server. I haven’t tested with it off and on thoroughly, but I saw some inconsistency when it was disabled. For the record, it’s on by default.
You can check with:
::*> cifs options show -fields is-use-junctions-as-reparse-points-enabled
Give it a try and let me know how it works for you in the comments!