XCP SMB/CIFS support available!

If you’re not familiar with what XCP is, I covered it in a previous blog post, Migrating to ONTAP – Ludicrous speed! as well as in the XCP podcast. Basically, it’s a super-fast way to scan and migrate data.

One of the downsides of the tool was the fact that it only supported NFSv3 migrations, which also meant it couldn’t handle NTFS style ACLs. Doing that would require a SMB/CIFS supported version of XCP. Today, we get that with XCP SMB/CIFS 1.0:

https://mysupport.netapp.com/tools/download/ECMLP2357425DT.html?productID=62115&pcfContentID=ECMLP2357425

XCP for SMB/CIFS supports the following:

“show” Displays information about the CIFS shares of a system
“scan”  Reads all files and directories found on a CIFS share and build assessment reports
“copy”  Recursively copies everything from source to destination
“sync”  Performs multiple incremental syncs from source to target
“verify”  Verifies that the target state matches the source, including attributes and NTFS ACLs
“activate”  Activates the XCP license on Windows hosts
“help”     Displays detailed information about XCP commands and options

 

Right now, it’s CLI only, but be on the lookout for a GUI version.

“Installing” XCP on Windows

XCP in Windows is a simple executable file that runs via the cmd or a PowerShell window. One of the pre-requisites for the software includes Microsoft Visual C++ Redistributable for Visual Studio 2017. If you don’t install this, trying to run the program will result in an error that calls out a specific DLL that isn’t registered.

When I copied the file to my Windows host, I created a new directory called “C:\XCP.” You can put that directory anywhere. To run the utility in CMD, you can either navigate to the directory and run “xcp” or add the directory to your system paths to run from anywhere.

For example:

env-windows-path

XCP-path

Once that’s done, run XCP from any location:

cifs-xcp

cifs-xcp-ps.png

Licensing XCP

XCP is a licensed feature. That doesn’t mean you have to pay for it; the license is only used for tracking purposes. But you do have to apply a license. In Windows, that’s pretty easy.

  1. Download a license from xcp.netapp.com
  2. Copy the license into the C:\NetApp\XCP folder
  3. Run “xcp activate”

xcp-license.png

XCP show

The command “xcp show \\server” can give some useful information for an ONTAP SMB/CIFS server, such as:

  • Available shares
  • Capacity (used and available)
  • Current connections
  • Folder path
  • Share attributes and permissions

This output is a good way to get an overall look at what is available on a server.

cifs-xcp-show.png

XCP scan

XCP has a number of useful scanning features. These include:

PS C:\XCP> xcp help scan

usage: xcp scan [-h] [-v] [-parallel <n>] [-match <filter>] [-preserve-atime]
 [-depth <n>] [-stats] [-l] [-ownership] [-du]
 [-fmt <expression>]
 source

positional arguments:
 source

optional arguments:
 -h, --help show this help message and exit
 -v increase debug verbosity
 -parallel <n> number of concurrent processes (default: <cpu-count>)
 -match <filter> only process files and directories that match the filter
 (see `xcp help -match` for details)
 -preserve-atime restore last accessed date on source
 -depth <n> limit the search depth
 -stats print tree statistics report
 -l detailed file listing output
 -ownership retrieve ownership information
 -du summarize space usage of each directory including
 subdirectories
 -fmt <expression> format file listing according to the python expression
 (see `xcp help -fmt` for details)

I scanned my “shared” directory with the -stats option and it was able to scan over 60,000 files in 31 seconds and gave me the following stats:

== Maximum Values ==
 Size Depth Namelen Dirsize
 2.02KiB 5 15 100

== Average Values ==
 Size Depth Namelen Dirsize
 25.6 5 6 6

== Top File Extensions ==
 .py
 50003 1

== Number of files ==
 empty <8KiB 8-64KiB 64KiB-1MiB 1-10MiB 10-100MiB >100MiB
 3 50001

== Space used ==
 empty <8KiB 8-64KiB 64KiB-1MiB 1-10MiB 10-100MiB >100MiB
 0 1.22MiB 0 0 0 0 0

== Directory entries ==
 empty 1-10 10-100 100-1K 1K-10K >10k
 2 10004 101

== Depth ==
 0-5 6-10 11-15 16-20 21-100 >100
 60111

== Modified ==
 >1 year >1 month 1-31 days 1-24 hrs <1 hour <15 mins future
 60111

== Created ==
 >1 year >1 month 1-31 days 1-24 hrs <1 hour <15 mins future
 60111

Total count: 60111
Directories: 10107
Regular files: 50004
Symbolic links:
Junctions:
Special files:
Total space for regular files: 1.22MiB
Total space for directories: 0
Total space used: 1.22MiB
60,111 scanned, 0 errors, 31s

When I increased the parallel threads to 8, it finished in 18 seconds:

PS C:\XCP> xcp scan -stats -parallel 8 \\demo\shared

Total count: 60111
Directories: 10107
Regular files: 50004
Symbolic links:
Junctions:
Special files:
Total space for regular files: 1.22MiB
Total space for directories: 0
Total space used: 1.22MiB
60,111 scanned, 0 errors, 18s

XCP copy

With xcp copy, I can copy SMB/CIFS data with or without ACLs at a much faster rate than simple robocopy. Keep in mind that with this version of XCP, it doesn’t have BACKUP OPERATOR rights, so you’d need to run the utility as an admin user on both source and destination.

In the following example, I used robocopy to copy the same dataset as XCP to a NetApp FlexGroup volume.

Robocopy to FlexGroup results (~20-30 minutes)

         Total Copied Skipped Mismatch FAILED Extras
 Dirs :  10107  10106       1        0      0      0
 Files : 50004  50004       0        0      0      0
 Bytes : 1.21m  1.21m       0        0      0      0
 Times : 0:19:01 0:13:11 0:00:00 0:05:50

Speed : 1615 Bytes/sec.
 Speed : 0.092 MegaBytes/min.

UPDATE: Someone asked if the above robocopy run was done with the /MT flag, which would be a more fair apples to apples comparison, since XCP does multithreading. It wasn’t. The syntax used was:

PS C:\XCP> robocopy /S /COPYALL source destination

So, I re-ran it using MT:8 and with an empty FlexGroup after restoring the base snapshot and converting the security style to NTFS to ensure the ACLs come over as well. The multithreading of robocopy cut the time to completion roughly in half.

Robocopy /MT to FlexGroup results (~8-9 minutes)

 PS C:\XCP> robocopy /S /COPYALL /MT:8 \\demo\shared \\demo\flexgroup\robocopyMT

-------------------------------------------------------------------------------
 ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : Tue Aug 22 20:32:54 2017

Source : \\demo\shared\
 Dest : \\demo\flexgroup\robocopyMT\

Files : *.*

Options : *.* /S /COPYALL /MT:8 /R:1000000 /W:30
------------------------------------------------------------------------------
Total Copied Skipped Mismatch FAILED Extras
 Dirs : 10107 10106 1 0 0 0
 Files : 50004 50004 0 0 0 0
 Bytes : 1.21 m 1.21 m 0 0 0 0
 Times : 0:35:21 0:06:23 0:00:00 0:01:59

Ended : Tue Aug 22 20:41:18 2017

Then I re-ran the XCP to FlexGroup by restoring the baseline snapshot and then making sure the security style of the volume was NTFS. (It was UNIX before, which would have affected ACLs and overall speed). But, the run still held within 4 minutes. So, we’re looking at 2x as fast as robocopy with a small 60k file and folder workload. In addition, the host I’m using is a Windows 7 client VM with a 1GB network connection and not a ton of power behind it. XCP works best with more robust hardware.

win7-info

XCP to FlexGroup results – NTFS security style (~4 minutes!)

PS C:\XCP> xcp copy -parallel 8 \\demo\shared \\demo\flexgroup\XCP
1,436 scanned, 0 errors, 0 skipped, 0 copied, 0 (0/s), 5s
4,381 scanned, 0 errors, 0 skipped, 507 copied, 12.4KiB (2.48KiB/s), 10s
5,426 scanned, 0 errors, 0 skipped, 1,882 copied, 40.5KiB (5.64KiB/s), 15s
7,431 scanned, 0 errors, 0 skipped, 3,189 copied, 67.4KiB (5.37KiB/s), 20s
8,451 scanned, 0 errors, 0 skipped, 4,537 copied, 96.1KiB (5.75KiB/s), 25s
9,651 scanned, 0 errors, 0 skipped, 5,867 copied, 123KiB (5.31KiB/s), 30s
10,751 scanned, 0 errors, 0 skipped, 7,184 copied, 150KiB (5.58KiB/s), 35s
12,681 scanned, 0 errors, 0 skipped, 8,507 copied, 178KiB (5.44KiB/s), 40s
13,891 scanned, 0 errors, 0 skipped, 9,796 copied, 204KiB (5.26KiB/s), 45s
14,861 scanned, 0 errors, 0 skipped, 11,136 copied, 232KiB (5.70KiB/s), 50s
15,966 scanned, 0 errors, 0 skipped, 12,464 copied, 259KiB (5.43KiB/s), 55s
18,031 scanned, 0 errors, 0 skipped, 13,784 copied, 287KiB (5.52KiB/s), 1m0s
19,056 scanned, 0 errors, 0 skipped, 15,136 copied, 316KiB (5.80KiB/s), 1m5s
20,261 scanned, 0 errors, 0 skipped, 16,436 copied, 342KiB (5.21KiB/s), 1m10s
21,386 scanned, 0 errors, 0 skipped, 17,775 copied, 370KiB (5.65KiB/s), 1m15s
23,286 scanned, 0 errors, 0 skipped, 19,068 copied, 397KiB (5.36KiB/s), 1m20s
24,481 scanned, 0 errors, 0 skipped, 20,380 copied, 424KiB (5.44KiB/s), 1m25s
25,526 scanned, 0 errors, 0 skipped, 21,683 copied, 451KiB (5.35KiB/s), 1m30s
26,581 scanned, 0 errors, 0 skipped, 23,026 copied, 479KiB (5.62KiB/s), 1m35s
28,421 scanned, 0 errors, 0 skipped, 24,364 copied, 507KiB (5.63KiB/s), 1m40s
29,701 scanned, 0 errors, 0 skipped, 25,713 copied, 536KiB (5.70KiB/s), 1m45s
30,896 scanned, 0 errors, 0 skipped, 26,996 copied, 561KiB (5.15KiB/s), 1m50s
31,911 scanned, 0 errors, 0 skipped, 28,334 copied, 590KiB (5.63KiB/s), 1m55s
33,706 scanned, 0 errors, 0 skipped, 29,669 copied, 617KiB (5.52KiB/s), 2m0s
35,081 scanned, 0 errors, 0 skipped, 30,972 copied, 644KiB (5.44KiB/s), 2m5s
36,116 scanned, 0 errors, 0 skipped, 32,263 copied, 671KiB (5.30KiB/s), 2m10s
37,201 scanned, 0 errors, 0 skipped, 33,579 copied, 698KiB (5.48KiB/s), 2m15s
38,531 scanned, 0 errors, 0 skipped, 34,898 copied, 726KiB (5.65KiB/s), 2m20s
40,206 scanned, 0 errors, 0 skipped, 36,199 copied, 753KiB (5.36KiB/s), 2m25s
41,371 scanned, 0 errors, 0 skipped, 37,507 copied, 780KiB (5.39KiB/s), 2m30s
42,441 scanned, 0 errors, 0 skipped, 38,834 copied, 808KiB (5.63KiB/s), 2m35s
43,591 scanned, 0 errors, 0 skipped, 40,161 copied, 835KiB (5.47KiB/s), 2m40s
45,536 scanned, 0 errors, 0 skipped, 41,445 copied, 862KiB (5.31KiB/s), 2m45s
46,646 scanned, 0 errors, 0 skipped, 42,762 copied, 890KiB (5.56KiB/s), 2m50s
47,691 scanned, 0 errors, 0 skipped, 44,052 copied, 916KiB (5.30KiB/s), 2m55s
48,606 scanned, 0 errors, 0 skipped, 45,371 copied, 943KiB (5.45KiB/s), 3m0s
50,611 scanned, 0 errors, 0 skipped, 46,518 copied, 967KiB (4.84KiB/s), 3m5s
51,721 scanned, 0 errors, 0 skipped, 47,847 copied, 995KiB (5.54KiB/s), 3m10s
52,846 scanned, 0 errors, 0 skipped, 49,138 copied, 1022KiB (5.32KiB/s), 3m15s
53,876 scanned, 0 errors, 0 skipped, 50,448 copied, 1.02MiB (5.53KiB/s), 3m20s
55,871 scanned, 0 errors, 0 skipped, 51,757 copied, 1.05MiB (5.42KiB/s), 3m25s
57,011 scanned, 0 errors, 0 skipped, 53,080 copied, 1.08MiB (5.52KiB/s), 3m30s
58,101 scanned, 0 errors, 0 skipped, 54,384 copied, 1.10MiB (5.39KiB/s), 3m35s
59,156 scanned, 0 errors, 0 skipped, 55,714 copied, 1.13MiB (5.57KiB/s), 3m40s
60,111 scanned, 0 errors, 0 skipped, 57,049 copied, 1.16MiB (5.52KiB/s), 3m45s
60,111 scanned, 0 errors, 0 skipped, 58,483 copied, 1.19MiB (6.02KiB/s), 3m50s
60,111 scanned, 0 errors, 0 skipped, 59,907 copied, 1.22MiB (5.79KiB/s), 3m55s
60,111 scanned, 0 errors, 0 skipped, 60,110 copied, 1.22MiB (5.29KiB/s), 3m56s

XCP sync and verify

Sync and verify can be used during data migrations to ensure the source and target match up before cutting over. These use the same multi-processing capabilities as copy, so this should also be fast. Keep in mind that sync could also potentially be used to do incremental backups using XCP!

xcp-verify.png

New Technical Report – Electronic Design Automation (EDA) Best Practices

eda-logo

With the introduction of FlexGroup volumes in ONTAP 9.1, I mention that one of the sweet spots for FlexGroup volume use cases is the EDA space, due to the high ingest and large number of files.

As such, I’ve written up a new TR for EDA best practices that can be found here:

http://www.netapp.com/us/media/tr-4617.pdf

What is EDA?

EDA stands for “Electronic Design Automation.” Essentially, it refers to software tools for designing electronic systems such as integrated circuits and printed circuit boards. The tools work together in a design flow that chip designers use to design and analyze entire semiconductor chips. Since a modern semiconductor chip can have billions of components, EDA tools are essential for their design. Here’s a list of EDA companies for reference:

https://en.wikipedia.org/wiki/Electronic_design_automation

Feel free to send feedback to the DL in the doc, or post in the comments here.

NVMe… so hot right now.

nvme-sohot

Recently, our good friends* at The Register wrote an article about NetApp’s NVMe strategy and how ONTAP will eventually work the technology into its architecture over time, rather than force-feeding customers NVMe.

* Good friends because they’re currently being nice to us. 😛

From the article:

Faster fabric interconnects such as 25/50/100Gbit Ethernet, 32Gbit/s Fibre Channel and NVME-over-Fabrics (NVMe-oF) were bringing network access speeds up to better access the faster media types coming. NetApp will initially use the new media selectively, then scale its adoption and fully integrate it, leading to broader adoption and optimised media.

NetApp Chief Evangelist, Jeff Baxter (@baxontap), recently spoke about NetApp’s NVMe direction at the Flash Memory Summit. You can view that here:

https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170810_Keynote14_NetApp.pdf

What is NVMe?

A common misconception about NVMe (the “e” stands for express!) is that it’s a *thing,* like a specific type of drive. Rather, it’s a way to connect things to faster, bigger pipes. Think of it as a way to replace SAS adapters. Some think it will change the world. Others think it’s just another eventuality in the storage industry. Did people get this excited about SAS adapters?

We attempt to “demystify” NVMe in Episode 72 of the Tech ONTAP podcast:

We also discuss it with Jeff Steiner (@tweetofsteiner) in this Episode 79:

One under the radar move towards a NVMe future was the acquisition of PlexiStor, which was already ahead of the curve with software designed to take advantage of the benefits of NVMe, as well as Storage Class Memory. How NetApp integrates PlexiStor’s intellectual property in the future remains to be seen…

Does NetApp currently use NVMe?

Actually, yes! But the way we use it is in an onboard caching mechanism, currently. For example, the new FAS26xx series uses an onboard 1TB NVMe attached read cache, along with other systems. NetApp CSE Keith Aasen (@keith_aasen) does a great job breaking down the whys and hows of that in this blog:

NMVe – A Step Into The Future

NetApp A-Team member John Woodall (@John_Woodall) also gives a nice breakdown of NetApp and NVMe here:

NetApp and NVMe – The Rest of the Story

NetApp Principal Architect Andy Grimes (@Andy_NTAP_Flash) discusses NVMe here with John Woodall:

As with anything new and exciting, stay tuned for more possible announcements concerning NVMe. Perhaps even at this year’s NetApp Insight conference.

If you want to stay up to date on the latest in NVMe at NetApp, be sure to bookmark the new NetApp NVMe landing page!

http://www.netapp.com/us/info/nvme.aspx

Non-disruptively upgrading ONTAP

Be sure to also check out:

How to Perform Continuous ONTAP Upgrades Without Sacrificing IT Stability

A while back, I wrote about why you’d want to upgrade to ONTAP 9.2, especially if you are using ONTAP for VMware environments. I mentioned the ability to do non-disruptive upgrades as a motivating factor.

Then, someone asked in the comments for a post on performing an upgrade… Unfortunately, the systems I have that can be upgraded to 9.2 is already on 9.2 (because 9.2 is awesome). The other systems I have are FAS3270s, which are not supported with 9.2. But, I’ll still show you how to get there. I just can’t give you screenshots or video of me doing the actual upgrade.

Before you start

Whenever you want to upgrade any software, you should do some planning. NetApp provides some tools to help with that, such as the Interoperability Matrix and Upgrade Advisor.

From the KB:

The Data ONTAP Upgrade Advisor is a tool within My AutoSupport that will provide a plan with steps and commands to successfully perform a Data ONTAP upgrade or revert. The plan also contains context-sensitive links to the following:

  • Pre and post upgrade checks
  • Issues and risk exposures
  • Data ONTAP Release Notes
  • Data ONTAP binaries
  • Links to shelf module, system, and diagnostic FW

Note: AutoSupport must be enabled and working for the tool to work. Data ONTAP Upgrade Advisor  uses the information within the latest AutoSupport available from the system to generate the plan. To enable or configure AutoSupport, visit the AutoSupport page on the NetApp Support site.

Interoperability Matrix (IMT)

Start with this first to save yourself some time and headaches. Why try to generate an upgrade plan for your systems if you don’t even know what versions are supported for your platform? In the new IMT, you have several choices in how to search:

imt-search

Choose the one that works best for you.

My AutoSupport

Keep in mind that there is a new My AutoSupport interface, so be sure to check out this short video overview:

Finding Upgrade Advisor

When you open up a NetApp system in MyAutoSupport, there are two places you can access Upgrade Advisor.

The side menu:

upgrade-advisor-menu.png

And the “Upgrade Recommendation” section:

upgrade-advisor-button

Once you click on either, you will be taken to a screen where it will auto-populate the cluster’s serial numbers and allow you to start the upgrade process. From this page, click next:

upgrade-advisor-addsn

The next page will take you to a screen that shows your your systems. From here, you can deviate from the recommended version and select your desired ONTAP version.

upgrade-advisor-change.png

Click on the hyperlinked target version and choose your desired version. Click the check box to apply to all systems.

upgrade-advisor-target.png

After the version is selected, click next.

The following page allows you to specify what type of upgrade you want (ANDU is non-disruptive; DU is disruptive), as well as if you want to generate a revert plan, what file format, etc. Once you’re done, select “Generate” and a plan will be emailed to you.

upgrade-advisor-generate.png

In the case of my system, I can’t upgrade straight from 8.3.2 to 9.2. I have to go to 9.1 first. Upgrade Advisor will warn me, but ideally I’d have checked the Interoperability Matrix first.

upgrade-advisor-unsupport.png

I click previous, select 9.1P6 (the current latest 9.1 patch release) and click next and then “Generate.”

I’ll see a green box telling me the request was successful. Then I can navigate to t he “Upgrade Request Status” page to see the progress.

ua-request-status

From here, I can download the plan or wait for the email to arrive. The email will send a zip file, so if your email servers block .zip files, use the download process above.

The zip file will have the PDF or XLS versions of the upgrade and revert plans for each system you requested. These files will have a slew of pre-upgrade checks for you to perform. Once those are complete, you can begin your non-disruptive upgrade, as per the guide.

Here’s a nifty Datalink video of upgrades and best practices:

You can also get more info upgrading from 8.3.x to 9.x in this post:

https://community.netapp.com/t5/Data-ONTAP-Discussions/Upgrading-Clustered-Data-ONTAP-8-3x-to-ONTAP-9-1-Using-Automated-Nondisruptive/td-p/128391

While that post shows upgrading to 9.1, the same process works for 9.2.

Using NFSv4.x ACLs with NFSv3 in NetApp ONTAP? You betcha!

One thing I’ve come to realize from being in IT so long is that you should be constantly learning new things. If you aren’t, it’s not because you’re smart or because you know everything; it’s because you’re stagnating.

So, I was not surprised when I heard it was possible to apply NFSv4.x ACLs to files and folders and then mount them via NFSv3 and have the ACLs still work! I already knew that you could do audit ACEs from NFSv4.x for NFSv3 (covered in TR-4067), but had no idea this could extend into the permissions realm. If so, this solves a pretty big problem with NFSv3 in general, where your normal permissions are limited only to owner, group and then everyone else. That makes it hard to do any sort of granular access control for NFSv3 mounts, presents problems for some environments.

It also allows you to keep using NFSv3 for your workloads, whether for legacy application or general performance concerns. NFSv4.x has a lot of advantages over NFSv3, but if you don’t need stateful operations or the NFSv4.x features, or integrated locking, then you are safe to stay with NFSv3.

So, is it possible to use NFSv4.x ACLs with NFSv3 objects?

You betcha!

fargo-film-marge.jpg

The method for doing this is pretty straightforward.

  1. Configure and enable NFSv4.x in ONTAP and on your client
  2. Enable NFSv4.x ACL support in ONTAP
  3. Mount the export via NFSv4.x
  4. Apply the NFSv4.x ACLs
  5. Unmount and then remount the export using NFSv3 and test it out!

 

Configuring NFSv4.x

When you’re setting up NFSv4.x in an environment, there are a few things to keep in mind:

  • Client and NFS server support for NFSv4.x
  • NFS utilities installed on clients (for NFSv4.x functionality)
  • NFSv4.x configured on the client in idmapd.conf
  • NFSv4.x configured on the server in ONTAP (ACLS allowed)
  • Export policies and rules configured in ONTAP
  • Ideally, a name service server (like LDAP) to negotiate the server/client conversation of user identities

One of the reasons NFS4.x is more secure than NFSv3 is the use of user ID strings (such as user@domain.com) to help limit cases of user spoofing in NFS conversations. This ID string is required to be case-sensitive. If the string doesn’t match on both client and server, then the NFSv4.x mounts will get squashed to the defined “nobody” user in the NFSv4.x client. One of the more common issues seen with NFSv4.x mounts is the “nobody:nobody” user and group on files and folders. One of the most common causes of this is when a domain string is mismatched on the client and server.

In a client that domain string is defined in the idmapd.conf file. Sometimes, it will default to the DNS domain. In ONTAP, the v4-id-domain string should be configured to the same value on the client to provide proper NFSv4.x authentication.

Other measures, such as Kerberos encryption, can help lock the NFS conversations down further. NFSv4.x ACLs are a way to ensure that files and folders are only seen by those entities that have been granted access and is considered to be authorization, or, what you are allowed to do once you authenticate. For more complete steps on setting up NFSv4.x, see TR-4067 and TR-4073.

However, we’re only setting up NFSv4.x to allow us to configure the ACLs…

What are NFSv4.x ACLs?

NFSv4.x ACLs are a way to apply granular permissions to files and folders in NFS outside of the normal “read/write/execute” of NFSv3, and across more objects than simple “owner/group/everyone.” NFSv4.x ACLs allow administrators to set permissions for multiple users and groups on the same file or folder and treat NFS ACLs more like Windows ACLs. For more information on NFSv4.x ACLs, see:

http://wiki.linux-nfs.org/wiki/index.php/ACLs

https://linux.die.net/man/5/nfs4_acl

http://www.netapp.com/us/media/tr-4067.pdf

NFSv3 doesn’t have this capability by default. The only way to get more granular ACLs in NFSv3 natively is to use POSIX ACLs, which ONTAP doesn’t support.

Once you’ve enabled ACLs in ONTAP (v4.0-acl and/or v4.1-acl options), you can mount an NFS export via NFSv4.x and start applying NFSv4.x ACLs.

In my environment, I mounted a homedir volume and then set up an ACL on a file owned by root for a user called “prof1” using nfs4_setfacl -e (which allows you to edit a file rather than have to type in a long command).

[root@centos7 /]# mount demo:/home /mnt
[root@centos7 /]# mount | grep mnt
demo:/home on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.193.67.225,local_lock=none,addr=10.193.67.237)

The file lives in the root user’s homedir. The root homedir is set to 755, which means anyone can read them, but no one but the owner (root) can write to them.

drwxr-xr-x 2 root root 4096 Jul 13 10:42 root

That is, unless, I set NFSv4.x ACLs to allow a user full control:

[root@centos7 mnt]# nfs4_getfacl /mnt/root/file
A::prof1@ntap.local:rwaxtTnNcCy
A::OWNER@:rwaxtTnNcCy
A:g:GROUP@:rxtncy
A::EVERYONE@:rxtncy

I can also see those permissions from the ONTAP CLI:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: NFSV4 Security Descriptor
 Control:0x8014
 DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 ALLOW-OWNER@-0x1601bf
 ALLOW-GROUP@-0x1200a9-IG
 ALLOW-EVERYONE@-0x1200a9

I can also expand the mask to translate the hex:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file -expand-mask true

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: 0x20
 ...0 .... .... .... = Offline
 .... ..0. .... .... = Sparse
 .... .... 0... .... = Normal
 .... .... ..1. .... = Archive
 .... .... ...0 .... = Directory
 .... .... .... .0.. = System
 .... .... .... ..0. = Hidden
 .... .... .... ...0 = Read Only
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: NFSV4 Security Descriptor
 Control:0x8014

1... .... .... .... = Self Relative
 .0.. .... .... .... = RM Control Valid
 ..0. .... .... .... = SACL Protected
 ...0 .... .... .... = DACL Protected
 .... 0... .... .... = SACL Inherited
 .... .0.. .... .... = DACL Inherited
 .... ..0. .... .... = SACL Inherit Required
 .... ...0 .... .... = DACL Inherit Required
 .... .... ..0. .... = SACL Defaulted
 .... .... ...1 .... = SACL Present
 .... .... .... 0... = DACL Defaulted
 .... .... .... .1.. = DACL Present
 .... .... .... ..0. = Group Defaulted
 .... .... .... ...0 = Owner Defaulted

DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .1.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...1 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...1 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .1.. = Append
 .... .... .... .... .... .... .... ..1. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-OWNER@-0x1601bf
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .1.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...1 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...1 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .1.. = Append
 .... .... .... .... .... .... .... ..1. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-GROUP@-0x1200a9-IG
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .0.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...0 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...0 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .0.. = Append
 .... .... .... .... .... .... .... ..0. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-EVERYONE@-0x1200a9
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .0.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...0 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...0 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .0.. = Append
 .... .... .... .... .... .... .... ..0. = Write
 .... .... .... .... .... .... .... ...1 = Read

In the above, I gave prof1 full control over the file. Then, I mounted via NFSv3:

[root@centos7 /]# mount -o nfsvers=3 demo:/home /mnt
[root@centos7 /]# mount | grep mnt
demo:/home on /mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.193.67.219,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.193.67.219)

When I become a user that isn’t on the NFSv4.x ACL, I can’t write to the file:

[root@centos7 /]# su student1
sh-4.2$ cd /mnt/root
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwxr-xr-x 1 root bin 0 Jul 13 10:23 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

sh-4.2$ touch file
touch: cannot touch ‘file’: Permission denied
sh-4.2$ rm file
rm: remove write-protected regular empty file ‘file’? y
rm: cannot remove ‘file’: Permission denied

When I change to the prof1 user, I have access to do whatever I want, even though the mode bit permissions in v3 say I can’t:

[root@centos7 /]# su prof1
sh-4.2$ cd /mnt/root
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwxr-xr-x 1 root bin 0 Jul 13 10:23 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

sh-4.2$ vi file
sh-4.2$ cat file
NFSv4ACLS!

When I do a chmod, however, nothing seems to change from the NFSv4 ACL for the user. I set 700 on the file, which shows up in NFSv3 mode bits:

sh-4.2$ chmod 700 file
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwx------ 1 root bin 11 Aug 11 09:58 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

But notice how the prof1 user still has full control:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 700
 UNIX Mode Bits in Text: rwx------
 ACLs: NFSV4 Security Descriptor
 Control:0x8014
 DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 ALLOW-OWNER@-0x1601bf
 ALLOW-GROUP@-0x120088-IG
 ALLOW-EVERYONE@-0x120088

This is because of an ONTAP option known as “ACL Preservation.”

ontap9-tme-8040::*> nfs show -vserver DEMO -fields v4-acl-preserve
vserver v4-acl-preserve
------- ---------------
DEMO enabled

When I set the option to enabled, the NFSv4.x ACLs will survive mode bit changes. If I disable the option, the ACLs get blown away when a chmod is done:

ontap9-tme-8040::*> nfs modify -vserver DEMO -v4-acl-preserve disabled

ontap9-tme-8040::*> nfs show -vserver DEMO -fields v4-acl-preserve
vserver v4-acl-preserve
------- ---------------
DEMO disabled


[root@centos7 root]# chmod 755 file

And the ACLs are wiped out:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: -

I’d personally recommend setting that option to “enabled” if you want to do v3 mounts with v4.x ACLs.

So, there you have it… a new way to secure your NFSv3 mounts!

Behind the Scenes: Episode 100 – XCP

Welcome to the Episode 100, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

group-4-2016

This week is our 100th episode! In true TechONTAP podcast fashion, we didn’t celebrate it at all.

Instead, we stuck to the tech and brought in Bogdan Minciu and Joshey Lazer of the XCP team to discuss XCP and the upcoming release that supports CIFS/SMB.

Also, check out the podcast episode on migrations (where we chat about XCP) and this XCP blog.

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

You can listen here:

 

Encrypt your NFS packets end to end with krb5p and ONTAP 9.2!

NFS has always had a running joke about security, with a play on the acronym stating that NFS was “Not For Security.”

With NFSv3 and prior, there was certainly truth to that, especially when NFS was mounted without Kerberos. But even using Kerberos in NFSv3 wasn’t necessarily secure, as it only was applied to the NFS packets and not the extraneous services like NLM, NSM, mountd, etc.

NFSv4.x improved NFS security greatly by implementing a single port, ACLs, ID domain names and more tightly integrated support for Kerberos, among other improvements. However, simple krb5 authentication by itself only encrypts the initial mounts and not the NFS packets themselves.

That’s where stronger Kerberos modes like krb5i and krb5p come into play. From the RedHat man pages:

sec=krb5 uses Kerberos V5 instead of local UNIX UIDs and GIDs to authenticate users.

sec=krb5i uses Kerberos V5 for user authentication and performs integrity checking of NFS operations using secure checksums to prevent data tampering.

sec=krb5p uses Kerberos V5 for user authentication, integrity checking, and encrypts NFS traffic to prevent traffic sniffing. This is the most secure setting, but it also involves the most performance overhead.

krb5p = privacy

The p in krb5p stands for “privacy,” and it does that by way of Kerberos encryption of the NFS conversation end-to-end, via the specified encryption strength. The strongest you can currently use is AES-256. ONTAP 9.0 and later supports krb5p and AES-256 encryption. Krb5p is similar to SMB3 encryption/signing and sealing in its functionality.

Krb5p is also similar to SMB3 encryption in its performance impact; doing encryption of thousands of packets is expensive and can create CPU bottlenecks, unless…

AES-NI Offloading

AES-NI offloading is a feature available on specific Intel CPUs that allow encryption processing to use hardware acceleration instructions to offload processing for encryption. This allows the encryption to be done separately to alleviate performance bottlenecks.

From Intel’s site:

Intel® AES New Instructions (Intel® AES NI) is a new encryption instruction set that improves on the Advanced Encryption Standard (AES) algorithm and accelerates the encryption of data in the Intel® Xeon® processor family and the Intel® Core™ processor family.

Comprised of seven new instructions, Intel® AES-NI gives your IT environment faster, more affordable data protection and greater security; making pervasive encryption feasible in areas where previously it was not.

ONTAP 9.1 provided support for AES-NI offloading for SMB3 encryption, which greatly improved performance. But krb5p offloading was only added as of ONTAP 9.2. If you plan on using the end-to-end encryption functionality in NFS with krb5p, use ONTAP 9.2 or later. For more information on what other features are in ONTAP 9.2, see the following post:

ONTAP 9.2RC1 is available!

Krb5p performance in ONTAP 9.0 vs. ONTAP 9.2

Krb5p support was added in ONTAP 9.0, but the performance was pretty awful, due to the lack of AES-NI support.

Here are some graphs using SIO with different flavors of Kerberos and AUTH_SYS in ONTAP 9.0. (All using NFSv4.1)

In ONTAP 9.0, krb5p wasn’t ever able to achieve above 12k IOPS for 4k reads in these SIO tests, and what it was able to achieve, it did it at some pretty severe latency. Krb5i did a little better, but krb5 and auth_sys performed way better.

Test environment was:

  • FAS8080 (AFF numbers coming soon)
  • 12 RHEL 6.7 clients

4K sequential reads in ONTAP 9.0:

krb5-ontap9-4k-read

Writes are even worse for krb5p in ONTAP 9.0 – we didn’t even get to 10k.

4K sequential writes in ONTAP 9.0:

krb5-ontap9-4k-write

For 8K sequential reads in ONTAP 9.0, latency is about the same. Fewer ops, but that’s because we’re doing the same amount of work in bigger I/O chunks.

8K sequential reads in ONTAP 9.0:

krb5-ontap9-8k-read.png

8K sequential writes in ONTAP 9.0:

krb5-ontap9-8k-write.png

NOTE: ONTAP 9.1 was not tested, but I’d expect similar performance, as we don’t do AES-NI offloading for NFS in that release.

ONTAP 9.2 Kerberos 5p Performance – Vastly improved

Now, let’s compare those same tests to ONTAP 9.2 with the AES-NI offloading and other performance enhancements. In the graphs below, there are a few things to point out.

  • Much more predictable performance for krb5i and krb5p as IOPS increase
  • Lower latency in 9.2 at high IOPS for krb5 than in 9.0
  • No real peak IOPS for krb5i/krb5p; these security flavors are able to keep up with sys and krb5 for sheer maximum IOPS
  • Sub millisecond latency for NFS at high IOPS (~50k) in most workloads, regardless of the security flavor
  • AES-NI offloading and NFS performance improvements in ONTAP 9.2 are pretty substantial

4K Sequential Reads in ONTAP 9.2:

krb5-ontap92-4k-read.png

4K Sequential Writes in ONTAP 9.2:

krb5-ontap92-4k-write.png

8K sequential reads in ONTAP 9.2:

krb5-ontap92-8k-read.png

8K sequential writes in ONTAP 9.2:

krb5-ontap92-8k-write.png

Conclusion

With ONTAP 9.2, you can now get enterprise class security with Kerberos 5p along with performance that doesn’t kill your workloads. If you’re doing NFS with any flavor of Kerberos, it makes a ton of sense to upgrade to ONTAP 9.2 to receive the performance benefits from AES-NI offloading. Keep in mind that upgrading ONTAP is non-disruptive to NFSv3, as it’s stateless, but will be slightly disruptive to CIFS/SMB and NFSv4.x workloads, due to the statefulness of the protocols.

Behind the Scenes: Episode 97 – ONTAP Analytics and Telemetry Service (OATS)

Welcome to the Episode 97, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

group-4-2016

This week on the podcast, we’re talking ONTAP Analytics in the cloud and the new OATS product with a crew of performance folks including Greg Keller (@kellergc, Sr. Director, Performance Engineering), Matt Hambrick (Director of Performance Engineering) and Fred Peiffer (Performance Characterization Engineer/Architect). Find out how OATS is using existing performance data and machine learning to teach a system how to analyze and resolve performance issues, as well as how you can use OATS for your environment.

You can find OATS in the AWS Marketplace at: https://aws.amazon.com/marketplace/pp/B072JRLP87?qid=1497912376310&sr=0-10&ref_=srh_res_product_title

For an example of how to use OATS, see the following ONTAP recipe: https://community.netapp.com/t5/Data-ONTAP-Discussions/ONTAP-Recipes-Correlate-EMS-Messages-and-Performance-Metrics-with-NetApp-OATS/td-p/132184

For a short video, see:

To contact the team, email ng-oats-info@netapp.com.

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

You can listen here:

“Alexa, play NetApp’s Podcast…”

Amazon’s Alexa has been in and out of the news, both for its usefulness, as well as for some flaws.

220px-hal9000-svg

One of the cool things about Alexa is the ability to create programs and automation that ties directly into the device, such as controlling other devices connected to the internet.

One of NetApp’s software developers, Jungsook Yang, has taken Alexa and created automation for it to play the latest Tech ONTAP podcast episodes!

From Jungsook:

The podcast skill is live. The invocation name is ‘Netapp podcast’.

You can say, “Alexa, ask netapp podcast to start”, “Alexa, ask netapp podcast to help.” “Alexa, open netapp podcast.

And say “Alexa, pause” or “Alexa, resume” to pause and resume.

Currently, the automatic feed doesn’t work, though. Until I fix that part, I have to manually update the code whenever a new feed is introduced.

Pretty cool stuff! The product is currently “beta” and will be undergoing improvements. If you have feedback, send it to podcast@netapp.com or leave a comment!

Thanks a ton to Jungsook for this innovation!

How to host FTP-accessed data in ONTAP

I’m an official ONTAP chef!

maxresdefault[1]

That’s right; I created an ONTAP recipe, which shows you how to use NetApp ONTAP storage systems to host data for FTP shares.

You can find it here:

https://community.netapp.com/t5/Data-ONTAP-Discussions/ONTAP-Recipes-Work-around-the-lack-of-FTP-support-in-ONTAP-with-CIFS-SMB-or-NFS/m-p/132757

Keep in mind that native FTP support is not in ONTAP, so for the foreseeable future, use FTP VMs and CIFS/NFS shares. You can also use a similar approach to hosting HTTP data.

If you absolutely need native FTP/HTTP services in ONTAP, 7-Mode will be the way to go.

If you have any questions about the recipe, leave a comment here or on the recipe.