If you use NetApp ONTAP to host NAS shares (CIFS or NFS) and have too many files and folders to count, then you know how challenging it can be to figure out file information in your environment in a quick, efficient and effective way.
This becomes doubly important when you are thinking of migrating NAS data from FlexVol volumes to FlexGroup volumes, because there is some work up front that needs to be done to ensure you size the capacity of the FlexGroup and its member volumes correctly. TR-4571 covers some of that in detail, but it basically says “know your average file size.” It currently doesn’t tell you *how* to do that (though it will eventually). This blog attempts to fill that gap.
XCP
I’ve written previously about XCP here:
Generally speaking, it’s been to tout the data migration capabilities of the tool. But, in this case, I want to highlight the “xcp scan” capability.
XCP scan allows you to use multiple, parallel threads to analyze an unstructured NAS share much more quickly than you could with basic tools like rsync, du, etc.
The NFS version of XCP also allows you to output this scan to a file (HTML, XML, etc) to generate a report about the scanned data. It even does the math for you and finds the largest (max) file size and average file size!
The command I ran to get this information was:
# xcp scan -md5 -stats -html SERVER:/volume/path > filename.html
That’s it! XCP will scan and write to a file. You can also get info about the top five file consumers (by number and capacity) by owner, as well as get some nifty graphs. (Pro tip: Managers love graphs!)
What if I only have SMB/CIFS data?
Currently, XCP for SMB doesn’t support output to HTML files. But that doesn’t mean you can’t have fun, too!
You can stand up a VM using CentOS or whatever your favorite Linux kernel is and use XCP for NFS to scan data – provided the client has the necessary access to do so and you can score an NFS license (even if it’s eval). XCP scans are read-only, so you shouldn’t have issues running them.
Just keep in mind the following:
NFS to shares that have traditionally been SMB/CIFS-only are likely NTFS security style. This means that the user you are accessing the data as (for example, root) should be able to map to a valid Windows user that has read access to the data. NFS clients that access NTFS security style volumes map to Windows users to figure out permissions. I cover that here:
Mixed perceptions with NetApp multiprotocol NAS access
You can check the volume security style in two ways:
To check if the user you are attempting to access the volume via NFS with maps to a valid and expected Windows user, use this CLI command from diag privilege:
::> set diag
::*> diag secd name-mapping show -node node1 -vserver DEMO -direction unix-win -name prof1
'prof1' maps to 'NTAP\prof1'
To see what Windows groups this user would be a member of (and thus would get access to files and folders that have those groups assigned), use this diag privilege command:
::*> diag secd authentication show-creds -node ontap9-tme-8040-01 -vserver DEMO -unix-user-name prof1
UNIX UID: prof1 <> Windows User: NTAP\prof1 (Windows Domain User)
GID: ProfGroup
Supplementary GIDs:
ProfGroup
group1
group2
group3
sharedgroup
Primary Group SID: NTAP\DomainUsers (Windows Domain group)
Windows Membership:
NTAP\group2 (Windows Domain group)
NTAP\DomainUsers (Windows Domain group)
NTAP\sharedgroup (Windows Domain group)
NTAP\group1 (Windows Domain group)
NTAP\group3 (Windows Domain group)
NTAP\ProfGroup (Windows Domain group)
Service asserted identity (Windows Well known group)
BUILTIN\Users (Windows Alias)
User is also a member of Everyone, Authenticated Users, and Network Users
Privileges (0x2080):
SeChangeNotifyPrivilege
If you want to run XCP as root and want it to have administrator level access, you can create a name mapping. This is what I have in my SVM:
::> vserver name-mapping show -vserver DEMO -direction unix-win
Vserver: DEMO
Direction: unix-win
Position Hostname IP Address/Mask
-------- ---------------- ----------------
1 - - Pattern: root
Replacement: DEMO\\administrator
To create a name mapping for root to map to administrator:
::> vserver name-mapping create -vserver DEMO -direction unix-win -position 1 -pattern root -replacement DEMO\\administrator
Keep in mind that backup software often has this level of rights to files and folders, and the XCP scan is read-only, so there shouldn’t be any issue. If you are worried about making root an administrator, create a new Windows user for it to map to (for example, DOMAIN\xcp) and add it to the Backup Operators Windows Group.
In my lab, I ran a scan on a NTFS security style volume called “xcp_ntfs_src”:
::*> vserver security file-directory show -vserver DEMO -path /xcp_ntfs_src
Vserver: DEMO
File Path: /xcp_ntfs_src
File Inode Number: 64
Security Style: ntfs
Effective Style: ntfs
DOS Attributes: 10
DOS Attributes in Text: ----D---
Expanded Dos Attributes: -
UNIX User Id: 0
UNIX Group Id: 0
UNIX Mode Bits: 777
UNIX Mode Bits in Text: rwxrwxrwx
ACLs: NTFS Security Descriptor
Control:0x8014
Owner:NTAP\prof1
Group:BUILTIN\Administrators
DACL - ACEs
ALLOW-BUILTIN\Administrators-0x1f01ff-OI|CI
ALLOW-DEMO\Administrator-0x1f01ff-OI|CI
ALLOW-Everyone-0x100020-OI|CI
ALLOW-NTAP\student1-0x120089-OI|CI
ALLOW-NTAP\student2-0x120089-OI|CI
I used this command and nearly 600,000 objects were scanned in 25 seconds:
# xcp scan -md5 -stats -html 10.x.x.x:/xcp_ntfs_src > xcp-ntfs.html
XCP 1.3D1-8ae2672; (c) 2018 NetApp, Inc.; Licensed to Justin Parisi [NetApp Inc] until Tue Sep 4 13:23:07 2018
126,915 scanned, 85,900 summed, 43.8 MiB in (8.75 MiB/s), 14.5 MiB out (2.89 MiB/s), 5s
260,140 scanned, 187,900 summed, 91.6 MiB in (9.50 MiB/s), 31.3 MiB out (3.34 MiB/s), 10s
385,100 scanned, 303,900 summed, 140 MiB in (9.60 MiB/s), 49.9 MiB out (3.71 MiB/s), 15s
516,070 scanned, 406,530 summed, 187 MiB in (9.45 MiB/s), 66.7 MiB out (3.36 MiB/s), 20s
Sending statistics...
594,100 scanned, 495,000 summed, 220 MiB in (6.02 MiB/s), 80.5 MiB out (2.56 MiB/s), 25s
594,100 scanned, 495,000 summed, 220 MiB in (8.45 MiB/s), 80.5 MiB out (3.10 MiB/s), 25s.
This was the resulting report:
Happy scanning!
