Workaround for Mac Finder errors when unzipping files in ONTAP

ONTAP allows you to mount volumes to other volumes in a Storage Virtual Machine, which provides a way for storage administrators to create their own folder structures across multiple nodes in a cluster. This is useful when you want to ensure the workload gets spread across nodes, but you can’t use FlexGroup volumes for whatever reason.

This graphic shows how that can work:

junctioned-volumes.png

In NAS environments, a client will ask for a file or folder location and ONTAP will re-direct the traffic to wherever that object lives. This is supposed to be transparent to the client, provided they follow standard NAS deployment steps.

However, not all NAS clients are created equal. Sometimes, Linux serves up SMB and will do things differently than Microsoft does. Windows also will do NFS, but it doesn’t entirely follow the NFS RFCs. So, occasionally, ONTAP doesn’t expect how a client handles something and stuff breaks.

Mac Finder

If you’ve ever used a Mac, you’ll know that the Finder can do somethings a little differently than the Terminal does. In this particular issue, we’ll focus on how Finder unzips files (when you double-click the file) in volumes that are mounted to other volumes in ONTAP.

One of our customers hit this issue, and after poking around a little bit, I figured out how to workaround the issue.

Here’s what they were doing:

  • SMB to Mac clients
  • Shares at the parent FlexVol level (ie, /vol1
  • FlexVols mounted to other FlexVols several levels deep (ie, /vol1/vol2/vol3)

When files are unzipped after accessing a share at a higher level and then drilling down into other folders (which are actually FlexVols mounted to other FlexVols), then unzipping in Finder via double-click fails.

When the shares are mounted at the same level as the FlexVol where the unzip is attempted, unzip works. When the Terminal is used to unzip, it works.

However, when your users refuse to use/are unable to use the Terminal and you don’t want to create hundreds of shares just to work around one issue, it’s an untenable situation.

So, I decided to dig into the issue…

Reproducing the issue

The best way to troubleshoot problems is to set up a lab environment and try to recreate the problem. This allows you freedom to gather logs, packet traces, etc. without bothering your customer or end user. So, I brought my trusty old 2011 MacBook running OS Sierra and mounted the SMB share in question.

These are the volumes and their junction paths:

DEMO inodes /shared/inodes
DEMO shared /shared

This is the share:

 Vserver: DEMO
 Share: shared
 CIFS Server NetBIOS Name: DEMO
 Path: /shared
 Share Properties: oplocks
 browsable
 changenotify
 show-previous-versions
 Symlink Properties: symlinks
 File Mode Creation Mask: -
 Directory Mode Creation Mask: -
 Share Comment: -
 Share ACL: Everyone / Full Control
 File Attribute Cache Lifetime: -
 Volume Name: shared
 Offline Files: manual
 Vscan File-Operations Profile: standard
 Maximum Tree Connections on Share: 4294967295
 UNIX Group for File Create: -

I turned up debug logging on the cluster (engage NetApp Support if you want to do this), got a packet trace on the Mac and reproduced the issue right away. Lucky me!

finder-error

I also tried a 3rd party unzip utility (Stuffit Expander) and it unzipped fine. So this was definitely a Finder/ONTAP/NAS interaction problem, which allowed me to focus on that.

Packet traces showed that the Finder was attempting to look for a folder called “.TemporaryItems/folders.501/Cleanup At Startup” but couldn’t find it – and couldn’t create it, apparently either. But it would created folders named “BAH.XXXX” instead, and they wouldn’t get cleaned up.

So, I thought, why not manually create the folder path, since it wasn’t able to do it on its own?

You can do this through Terminal, or via Finder. Keep in mind that the path above has “folders.501” – 501 is my uid, so check your users uid on the Mac and make sure the folder path is created using that uid. If you have multiple users that access the share, you may need to create multiple folders.xxx in .TemporaryItems.

If you do it via Finder, you may want to enable hidden files. I learned how to do that via this article:

https://ianlunn.co.uk/articles/quickly-showhide-hidden-files-mac-os-x-mavericks/

So I did that and then I unmounted the share and re-mounted, to make sure there wasn’t any weird cache issue lingering. You can check CIFS/SMB sessions, versions, etc with the following command, if you want to make sure they are closed:

cluster::*> cifs session show -vserver DEMO -instance

Vserver: DEMO

Node: node1
 Session ID: 16043510722553971076
 Connection ID: 390771549
 Incoming Data LIF IP Address: 10.x.x.x
 Workstation IP Address: 10.x.x.x
 Authentication Mechanism: NTLMv2
 User Authenticated as: domain-user
 Windows User: NTAP\prof1
 UNIX User: prof1
 Open Shares: 1
 Open Files: 1
 Open Other: 0
 Connected Time: 7m 49s
 Idle Time: 6m 2s
 Protocol Version: SMB3
 Continuously Available: No
 Is Session Signed: true
 NetBIOS Name: -
 SMB Encryption Status: unencrypted
 Connection Count: 1

Once I reconnected with the newly created folder path, double-click unzip worked perfectly!

Check it out yourself:

Note: You *may* have to enable the option is-use-junctions-as-reparse-points-enabled on your CIFS server. I haven’t tested with it off and on thoroughly, but I saw some inconsistency when it was disabled. For the record, it’s on by default.

You can check with:

::*> cifs options show -fields is-use-junctions-as-reparse-points-enabled

Give it a try and let me know how it works for you in the comments!

Advertisements

Issues installing Docker on OS 10.10 or later?

Today I was trying to install the Docker Toolbox on my Mac. It failed. Because I was able to fix it and did not see any other articles on how to do this that specifically referenced this app or issue, I decided to write it up. Because, community!

docker_toolbox_banner_icon

The installation appeared to work fine, but once I clicked the “Docker Terminal” icon, the terminal would launch with the following message:

Docker Machine is not installed. Please re-run the Toolbox Installer and try again.

Docker Machine installs by default to the /usr/local/bin directory in OS X. And when I tried to change that location in the install package, I didn’t have any luck.

That directory is locked down pretty tight (700 permissions, my user as the owner).

drwx------  24 parisi  wheel  816 Mar 25 10:52 bin

When I tried to open the directory up a bit and re-install, it would have the same issue. And, when I tried to cd directly into that directory, it either threw a permission denied or silently failed, even though I had allowed access:

$ sudo chmod 766 bin

$ ls -la
total 0
drwxr-xr-x   5 root    wheel  170 Mar 25 10:50 .
drwxr-xr-x@ 10 root    wheel  340 May  8  2015 ..
drwxr-xr-x   3 root    wheel  102 Apr 20  2015 Qt
drwxrw-rw-  24 parisi  wheel  816 Mar 25 10:52 bin
drwxr-xr-x   3 root    wheel  102 Mar 25 10:50 share

$ cd bin
-bash: cd: bin: Permission denied

$ sudo cd bin

$ pwd
/usr/local

And Docker commands failed:

$ docker
-bash: docker: command not found

Color me stumped.

So I turned to Google and found an article on Homebrew installations failing, but nothing specifically on Docker failing. I used the Homebrew workaround found in this article and it fixed my issue.

Here are the commands I ran:

$ sudo chown $(whoami):admin /usr/local && sudo chown -R $(whoami):admin /usr/local

Essentially, the command above does a recursive (-R) chown on the /usr/local directories as the logged in user (via whoami).

Before the change, /usr/local looked like this:

drwxr-xr-x     6 root  wheel    204 Mar 25 10:56 local

After the change:

drwxr-xr-x     6 parisi  admin    204 Mar 25 10:56 local

After that, I could run Docker commands:

$ pwd
/Users/parisi

$ docker
Usage: docker [OPTIONS] COMMAND [arg...]
       docker [ --help | -v | --version ]
A self-sufficient runtime for containers.

Options:
  --config=~/.docker              Location of client config files
  -D, --debug                     Enable debug mode
  -H, --host=[]                   Daemon socket(s) to connect to
  -h, --help                      Print usage
  -l, --log-level=info            Set the logging level
  --tls                           Use TLS; implied by --tlsverify
  --tlscacert=~/.docker/ca.pem    Trust certs signed only by this CA
  --tlscert=~/.docker/cert.pem    Path to TLS certificate file
  --tlskey=~/.docker/key.pem      Path to TLS key file
  --tlsverify                     Use TLS and verify the remote
  -v, --version                   Print version information and quit

...

And the Docker terminal starts correctly:

Creating CA: /Users/parisi/.docker/machine/certs/ca.pem
Creating client certificate: /Users/parisi/.docker/machine/certs/cert.pem
Running pre-create checks...
Creating machine...
(default) Copying /Users/parisi/.docker/machine/cache/boot2docker.iso to /Users/parisi/.docker/machine/machines/default/boot2docker.iso...
(default) Creating VirtualBox VM...
(default) Creating SSH key...
(default) Starting the VM...
(default) Check network to re-create if needed...
(default) Found a new host-only adapter: "vboxnet0"
(default) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: /usr/local/bin/docker-machine env default

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/

docker is configured to use the default machine with IP X.X.X.X
For help getting started, check out the docs at https://docs.docker.com

If you’re interested on more detail on the issue, check out the Homebrew blog, as well as this on System Integrity Protector (SIP):

https://support.apple.com/en-us/HT204899

Hopefully this helps someone else.

Another option, pointed out to me on Twitter, is to use the native Docker apps (still in beta):

https://blog.docker.com/2016/03/docker-for-mac-windows-beta/

If interested, I’ve written a couple other blogs on Docker.

TECH::Using NFS with Docker – Where does it fit in?

TECH::Docker + CIFS/SMB? That’s unpossible!