Welcome to the Episode 289, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week, we discuss the latest news in the NetApp/Rubrik partnership and how Rubrik works with Ben Kendall (Alliances Technical Partner Manager, Americas at Rubrik, https://www.linkedin.com/in/ben-kendall-6436609/) and PF Guglielmi (Alliances Field CTO @pfguglielmi) of Rubrik and Chris Maino (Manager, Americas Solutions Architects, chris.maino@netapp.com) of NetApp.
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Tech ONTAP Community
We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.
ONTAP is keeping the twice-yearly release cadence by providing spring and fall releases to keep up with the need for new and rapidly developing features. In addition, the cloud versions of ONTAP will use different release numbers. ONTAP 9.9 signifies the cloud release of ONTAP and ONTAP 9.9.1 is the on-prem release.
So, here’s a brief rundown on what’s new in ONTAP 9.9.1 (copied from the customer communiques that go out). You can also view the Feature Overview TR for ONTAP 9.9.1 here:
ONTAP System Manager enhanced UI. ONTAP System Manager 9.9.1 supports new ONTAP 9.9.1 capabilities including ASA Cluster expansion, SVM-DR support for FlexGroup volumes, and more. In addition to the new features, NetApp has made many enhancements to address customer feedback, such as improved capacity reporting, support for nested iGroups, UI performance optimizations, and much more.
SnapMirror® Business Continuity (General Availability)
For your SAN workloads, this feature enables flexible, cost-effective, instant failover for your most important business applications such as SAP, Oracle, and more. You have the flexibility to improve TCO by consolidating many different types of workloads onto your ONTAP based systems and then granularly deciding which critical SAN workloads you want supported with the continuous availability of SMBC. SMBC enables applications to fail over to a secondary copy of storage without requiring the application to reconnect to the storage or disrupting the users of that application. It expands your choices across the spectrum of NetApp solutions and partnerships that deliver business continuity, disaster recovery, backup and recovery, and Snapshot™ technology.
New MetroCluster IP enhancements.
Customers can maximize capacity and performance with scalability up to 8-node NetApp MetroCluster IP configurations. Also, ONTAP 9.9.1 RC1 supports routed layer 3 network configuration and up to 1,600 volumes per HA pair.
SnapLock/WORM Compliance
Storage efficiencies for SnapLock volumes – supports volume data compaction, cross volume or aggregate level deduplication (AFF only), continuous segment cleaning, and Temperature Sensitive Storage Efficiency.
Ransomware protection support for snapshot copies containing LUNs
FlexGroup Enhancements
SnapMirror topology (Cascade and Fanout) – replicate a FlexGroup from site A to site B to site C (cascade) or site A to sites B and C (fan-out)
Logical Space Accounting/Enforcement – Monitor and enforce based on the actual storage used before efficiencies are applied for more accurate chargebacks
All SAN Array Enhancements
Scalable out ASA clusters. Customers can leverage the power of ONTAP clustering with scale out up to 12 nodes of ASA systems. ASA clusters can include a mix of ASA system types.
Increase performance with NVMe-OF on ASA systems. With ONTAP 9.9.1 RC1, modernize and accelerate business applications nondisruptively with NVME-oF for ASA systems.
AFF to ASA in-place conversion. For your mission-critical SAN workloads running Unified AFF, take advantage of ASA benefits such as large LUN support (128TB) and symmetric active-active support with seamless AFF to ASA in-place conversion capability.
Single LUN performance improvement. With ONTAP 9.9.1 RC1, single LUN performance has been significantly improved compared to previous ONTAP releases. Ideal for simplifying deployments in virtual environments; for example, A800 can provide up to 400% more random read IOPS.
Data Protocol Enhancements
NFSv4.2 Security Labels (MAC labels) – MAC labels enable granular file access control with SE Linux, above and beyond Unix permissions and NFSv4 ACLs. The benefits provided by this feature include multilevel security for files, simplifying administration of sensitive NAS data in highly secure environments.
ONTAP S3 object tags for ObjectCreate and MultipartUpload calls – When a HEAD or GET is performed on an object, the user defined metadata and count of the number of tags will be returned as part of the HTTP header in the response. These tags allow you to better categorize your objects within ONTAP buckets for more robust data management and are compatible with applications that require the ability to create metadata and tags.
The concept behind a NetApp FlexGroup volume is that ONTAP presents a single large namespace for NAS data, while ONTAP handles the balance and placement of files and folders to the underlying FlexVol member volumes, rather than a storage administrator needing to manage that.
However, while not knowing/caring where files and folders live in your cluster is nice most of the time, there are occasions where you may need to figure out where a file or folder *actually* lives in the cluster – such as if a member volume has a large imbalance of capacity usage and you need to know what files need to be deleted/moved out of that volume. Previously, there’s been no real good way to do that, but thanks to the efforts of one of our global solutions architects (and one of the inventors of XCP), we now have a way and we don’t even need a treasure map.
What is NetApp XCP?
If you’re unfamiliar with NetApp XCP, it’s NetApp’s FREE copy utility/data move that also can be used to do file analytics. There are other use cases, too:
Because XCP can run in parallel from a client, it can perform tasks (such as find) much faster in high file count environments, so you’re not sitting around waiting for a command to finish for minutes/hours/days.
Since a FlexGroup is pretty much made for high file count environments, we’d want a way to quickly find files and their locations.
ONTAP NAS and File Handles
In How to identify a file or folder in ONTAP in NFS packet traces, I covered how to find inode information and a little bit about how ONTAP file handles are created/presented. The deep details aren’t super important here, but the general concept – that FlexGroup member volume information is stored in file handles that NFS can read – is.
Using that information and some parsing, there’s a Python script that can be used as an XCP plugin to translate file handles into member volume index numbers and present them in easy-to-read formats.
How to Use the “FlexGroup File Mapper” plugin with XCP
First of all, you’d need a client that has XCP installed. The version isn’t super important, but the latest release is generally the best release to use.
There are two methods we’ll use here to map files to member volumes.
Scan All Files/Folders in a FlexGroup and Map Them All to Member Volumes
Use a FlexGroup Member Volume Number and Find All Files in that Member Volume
To do this, I’ll use a FlexGroup that has ~2 million files.
First, copy the FlexGroup File Mapper plugin to the XCP host. The file name isn’t important, but when you run the XCP command, you’ll either want to specify the plugin’s location or run the command from the folder the plugin lives in.
On my XCP host, I have the plugin named fgid.py in /testXCP:
# ls -la | grep fgid.py
-rw-r--r-- 1 502 admin 1645 Mar 25 17:34 fgid.py
# pwd
/testXCP
Scan All Files/Folders in a FlexGroup and Map Them All to Member Volumes
In this case, we’ll map all files and folders to their respective FlexGroup member volumes.
You can also include -parallel (n) to control how many processes spin up to do this work and you can use > filename at the end to pipe the output to a file (recommended).
For example, scanning ~2 million files in this volume took just 37 seconds!
# xcp diag -run fgid.py scan -fmt '"{} {}".format(x, fgid(x))' 10.10.10.10:/FGNFS > FGNFS.txt
402,061 scanned, 70.6 MiB in (14.1 MiB/s), 367 KiB out (73.3 KiB/s), 5s
751,933 scanned, 132 MiB in (12.3 MiB/s), 687 KiB out (63.9 KiB/s), 10s
1.10M scanned, 193 MiB in (12.2 MiB/s), 1007 KiB out (63.6 KiB/s), 15s
1.28M scanned, 225 MiB in (6.23 MiB/s), 1.14 MiB out (32.6 KiB/s), 20s
1.61M scanned, 283 MiB in (11.6 MiB/s), 1.44 MiB out (60.4 KiB/s), 25s
1.91M scanned, 335 MiB in (9.53 MiB/s), 1.70 MiB out (49.5 KiB/s), 31s
2.00M scanned, 351 MiB in (3.30 MiB/s), 1.79 MiB out (17.4 KiB/s), 36s
Sending statistics…
Xcp command : xcp diag -run fgid.py scan -fmt "{} {}".format(x, fgid(x)) 10.10.10.10:/FGNFS
Stats : 2.00M scanned
Speed : 351 MiB in (9.49 MiB/s), 1.79 MiB out (49.5 KiB/s)
Total Time : 37s.
STATUS : PASSED
The file created was 120MB, though… that’s a LOT of text to sort through.
So, there’s another way to do this, right? Correct!
If I know the folder I want to filter, or even a matching of file names, I can use -match in the command. In this case, I want to find all folders named dir_33.
If I want to use pattern matching for file names (ie, I know I want all files with “moarfiles3” in the name), then I can do this using regex and/or wildcards. More examples can be found in the XCP user guides.
Here’s the command I used. It found 440,400 files with that pattern in 27s.
# xcp diag -run fgid.py scan -fmt '"{} {}".format(x, fgid(x))' -match "fnm('moarfiles3*')" 10.10.10.10:/FGNFS > moarfiles3_FGNFS.txt
507,332 scanned, 28,097 matched, 89.0 MiB in (17.8 MiB/s), 465 KiB out (92.9 KiB/s), 5s
946,796 scanned, 132,128 matched, 166 MiB in (15.4 MiB/s), 866 KiB out (80.1 KiB/s), 10s
1.31M scanned, 209,340 matched, 230 MiB in (12.8 MiB/s), 1.17 MiB out (66.2 KiB/s), 15s
1.73M scanned, 297,647 matched, 304 MiB in (14.8 MiB/s), 1.55 MiB out (77.3 KiB/s), 20s
2.00M scanned, 376,195 matched, 351 MiB in (9.35 MiB/s), 1.79 MiB out (48.8 KiB/s), 25s
Sending statistics…
Filtered: 444400 matched, 1556004 did not match
Xcp command : xcp diag -run fgid.py scan -fmt "{} {}".format(x, fgid(x)) -match fnm('moarfiles3*') 10.10.10.10:/FGNFS
Stats : 2.00M scanned, 444,400 matched
Speed : 351 MiB in (12.6 MiB/s), 1.79 MiB out (65.7 KiB/s)
Total Time : 27s.
And this is a sample of some of those entries (the file is 27MB):
I can also look for files over a certain size. In this volume, the files are all 4K in size; but in my TechONTAP volume, I have varying file sizes. In this case, I want to find all .wav files greater than 100MB. This command didn’t seem to pipe to a file for me, but the output was only 16 files.
# xcp diag -run fgid.py scan -fmt '"{} {}".format(x, fgid(x))' -match "fnm('.wav') and size > 500*M" 10.10.10.11:/techontap > TechONTAP_ep.txt
10.10.10.11:/techontap/Episodes/Episode 20x - Genomics Architecture/ep20x-genomics-meat.wav 4
10.10.10.11:/techontap/archive/combine.band/Media/Audio Files/ep104-webex.output.wav 5
10.10.10.11:/techontap/archive/combine.band/Media/Audio Files/ep104-mics.output.wav 3
10.10.10.11:/techontap/archive/Episode 181 - Networking Deep Dive/ep181-networking-deep-dive-meat.output.wav 6
10.10.10.11:/techontap/archive/Episode 181 - Networking Deep Dive/ep181-networking-deep-dive-meat.wav 2
Filtered: 16 matched, 7687 did not match
xcp command : xcp diag -run fgid.py scan -fmt "{} {}".format(x, fgid(x)) -match fnm('.wav') and size > 100M 10.10.10.11:/techontap
Stats : 7,703 scanned, 16 matched
Speed : 1.81 MiB in (1.44 MiB/s), 129 KiB out (102 KiB/s)
Total Time : 1s.
STATUS : PASSED
But what if I know that a member volume is getting full and I want to see what files are in that member volume?
Use a FlexGroup Member Volume Number and Find All Files in that Member Volume
In the case where I know what member volume needs to be addressed, I can use XCP to search using the FlexGroup index number. The index number lines up with the member volume numbers, so if the index number is 6, then we know the member volume is 6.
In my 2 million file FG, I want to filter by member 6, so I use this command, which shows there are ~95019 files in member 6:
# xcp diag -run fgid.py scan -match 'fgid(x)==6' -parallel 10 -l 10.10.10.10:/FGNFS > member6.txt
615,096 scanned, 19 matched, 108 MiB in (21.6 MiB/s), 563 KiB out (113 KiB/s), 5s
1.03M scanned, 5,019 matched, 180 MiB in (14.5 MiB/s), 939 KiB out (75.0 KiB/s), 10s
1.27M scanned, 8,651 matched, 222 MiB in (8.40 MiB/s), 1.13 MiB out (43.7 KiB/s), 15s
1.76M scanned, 50,019 matched, 309 MiB in (17.3 MiB/s), 1.57 MiB out (89.9 KiB/s), 20s
2.00M scanned, 62,793 matched, 351 MiB in (8.35 MiB/s), 1.79 MiB out (43.7 KiB/s), 25s
Filtered: 95019 matched, 1905385 did not match
Xcp command : xcp diag -run fgid.py scan -match fgid(x)==6 -parallel 10 -l 10.10.10.10:/FGNFS
Stats : 2.00M scanned, 95,019 matched
Speed : 351 MiB in (12.5 MiB/s), 1.79 MiB out (65.0 KiB/s)
Total Time : 28s.
STATUS : PASSED
When I check against the files-used for that member volume, it lines up pretty well:
And, if I choose, I can filter further with the sizes. Maybe I just want to see files in that member volume that are 4K or less (in this case, that’s all of them):
When you’re troubleshooting NFS issues, sometimes you have to collect a packet capture to see what’s going on. But the issue is, packet captures don’t really tell you the file or folder names. I like to use Wireshark for Mac and Windows, and regular old tcpdump for Linux. For ONTAP, you can run packet captures using this KB (requires NetApp login):
By default, Wireshark shows NFS packets like this ACCESS call. We see a FH, which is in hex, and then we see another filehandle that’s even more unreadable. We’ll occasionally see file names in the trace (like copy-file below), but if we need to find out why an ACCESS call fails, we’ll have difficulty:
Luckily, Wireshark has some built-in stuff to crack open those NFS file handles in ONTAP.
First, we’d want to set the NFS preferences. That’s done via Edit -> Preferences and then by clicking on “Protocols” in the left hand menu and selecting NFS:
Here, you’ll see some options that you can read more about by mousing over them:
I just select them all.
When we go to the packet we want to analyze, we can right click and select “Decode As…”:
This brings up the “Decode As” window. Here, we have “NFS File Handle Types” pre-populated. Double-click (none) under “Current” and you get a drop down menu. You’ll get some options for NFS, including…. ONTAP! In this case, since I’m using clustered ONTAP, I select ontap_gx_v3. (GX is what clustered ONTAP was before clustered ONTAP was clustered ONTAP):
If you click “OK” it will apply to the current session only. If you click “Save” it will keep those preferences every time.
Now, when the ACCESS packet is displayed, I get WAY more information about the file in question and they’re translated to decimal values.
Those still don’t mean a lot to us, but I’ll get to that.
Mapping file handle values to files in ONTAP
Now, we can use the ONTAP CLI and the packet capture to discern exactly what file has that ACCESS call.
Every volume in ONTAP has a unique identifier called a “Master Set ID” (or MSID). You can see the volume’s MSID with the following diag priv command:
If you know the volume name you’re troubleshooting, then that makes life easier – just use find in the packet details.
If you don’t, the MSID can be found in a packet trace in the ACCESS reply as the “fsid”:
You can then find the volume name and exported path with the MSID in the ONTAP CLI with:
cluster::*> set diag; vol show -vserver DEMO -msid 2163230318 -fields volume,junction-path
vserver volume junction-path
------- ------- -----------
DEMO vol2 /vol2
File and directory handles are constructed using that MSID, which is why each volume is considered a distinct filesystem. But we don’t care about that, because Wireshark figures all that out for us and we can use the ONTAP CLI to figure it out as well.
The pertinent information in the trace as it maps to the files and folders are:
Spin file id = inode number in ONTAP
Spin file unique id = file generation number
File id = inode number as seen by the NFS client
If you know the volume and file or folder’s name, you can easily find the inode number in ONTAP with this command:
::*> node run -node node2 inodepath -v FG2__0007 5292 Inode 5292 in volume FG2__0007 (fsid 0x1639) has 1 name. Volume UUID is: 87b14652-9685-11eb-81bf-00a0986b1223 [ 1] Primary pathname = /vol/FG2/copy-file-finder
There’s also a diag privilege command in ONTAP for that. The caveat is it can be dangerous to run, especially if you make a mistake in running it. (And when I say dangerous, I mean best case, it hangs your CLI session for a while; worst case, it panics the node.) If possible, use inodepath instead.
Here’s how we could use the volume name and inode number to find the file name. For a FlexVol volume, it’s simple:
cluster::*> vol explore -format inode -scope volname.inode -dump name
For example:
cluster::*> volume explore -format inode -scope files.15447 -dump name
name=/newapps/user1-file-smb
With a FlexGroup volume, however, it’s a little more complicated, as there are member volumes to take into account and there’s no easy way for ONTAP to discern which FlexGroup member volume has the file, since ONTAP inode numbers can be reused in different member volumes. This is because the file IDs presented to NFS clients are created using the inode numbers and things like the member volume’s MSID (which is different than the FlexGroup’s MSID).
To make this happen with volume explore, we’d be working in reverse – listing the contents of the volume’s files/folders, then using the inode number of the parent folder, listing those, etc. With high file count environments, this is basically an impossibility.
In that case, we’d need to use an NFS client to discover the file name associated with the inode number in question.
From the client, we have two commands to find an inode number for a file. In this case we know the file’s location and name:
# ls -i /mnt/client1/copy-file-finder
4133624749 /mnt/client1/copy-file-finder
In a packet trace, that inode number is “fileid” and found in REPLY calls, such as GETATTR:
If we only know the inode number (as if we got it from a packet trace), we can use the number on the client to find the file name. One way is with “find”:
Funny story about this post. Someone pointed out I had some broken links, so I went in and edited the links. When I clicked “publish” it re-posted the article, which was actually a pointer back to an old DatacenterDude article I wrote from 2015 – which no longer exists. So I started getting *more* pings about broken links and plenty of people seemed to be interested in the content. Thanks to the power of the Wayback Machine, I was able to resurrect the post and decided to do some modernization while I was at it.
Yesterday, I was speaking with a customer who is a cloud provider. They were discussing how to use NFSv4 with Data ONTAP for one of their customers. As we were talking, I brought up pNFS and its capabilities. They were genuinely excited about what pNFS could do for their particular use case. In the cloud, the idea is to remove the overhead of managing infrastructure, so most cloud architectures are geared towards automation, limiting management, etc. In most cases, that’s great, but for data locality in NAS environments, we need a way to make those operations seamless, as well as providing the best possible security available. That’s where pNFS comes in.
So, let’s talk about what pNFS is and in what use cases you may want to use it.
What is pNFS?
pNFS is “parallel NFS,” which is a little bit of a misnomer in ONTAP, as it doesn’t do parallel reads and writes across single files (i.e., striping). In the case of pNFS on Data ONTAP, NetApp currently supports file-level pNFS, so the object store would be a flexible volume on an aggregate of physical disks.
pNFS in ONTAP establishes a metadata path to the NFS server and then splits off the data path to its own dedicated path. The client works with the NFS server to determine which path is local to the physical location of the files in the NFS filesystem via DEVICEINFO and LAYOUTGETINFO metadata calls (specific to NFSv4.1 and later) and then dynamically redirects the path to be local. Think of it as ALUA for NFS.
The following graphic shows how that all takes place.
pNFS defines the notion of a device that is generated by the server (that is, an NFS server running on Data ONTAP) and sent to the client. This process helps the client locate the data and send requests directly over the path local to that data. Data ONTAP generates one pNFS device per flexible volume. The metadata path does not change, so metadata requests might still be remote. In a Data ONTAP pNFS implementation, every data LIF is considered an NFS server, so pNFS only works if each node owns at least one data LIF per NFS SVM. Doing otherwise negates the benefits of pNFS, which is data locality regardless of which IP address a client connects to.
The pNFS device contains information about the following:
Volume constituents
Network location of the constituents
The device information is cached to the local node for improved performance.
To see pNFS devices in the cluster, use the following command in advanced privilege:
cluster::> set diag cluster::*> vserver nfs pnfs devices cache show
pNFS Components
There are three main components of pNFS:
Metadata server
Handles all nondata traffic such as GETATTR, SETATTR, and so on
Responsible for maintaining metadata that informs the clients of the file locations
Located on the NetApp NFS server and established via the mount point
Data server
Stores file data and responds to READ and WRITE requests
To check if pNFS is in use, you can run statistics counters to check for “pnfs_layout_conversions” counters. If the number of pnfs_layout_conversions are incrementing, then pNFS is in use. Keep in mind that if you try to use pNFS with a single network interface, the data layout conversations won’t take place and pNFS won’t be used, even if it’s enabled.
cluster::*> statistics start -object nfsv4_1_diag
cluster::*> statistics show -object nfsv4_1_diag -counter pnfs_layout_conversions
Counter Value -------------------------------- -------------------------------- pnfs_layout_conversions 4053
Gotta keep ’em separated!
One thing that is beneficial about the design of pNFS is that the metadata paths are separated from the read/write paths. Once a mount is established, the metadata path is set on the IP address used for mount and does not move without manual intervention. In Data ONTAP, that path could live anywhere in the cluster. (Up to 24 physical nodes with multiple ports on each node!)
That buys you resiliency, as well as flexibility to control where the metadata will be served.
The data path, however, will only be established on reads and writes. That path is determined in conversations between the client and server and is dynamic. Any time the physical location of a volume changes, the data path changes automatically, without need to intervene by the clients or the storage administrator. So, unlike NFSv3 or even NFSv4, you no longer would need to break the TCP connection to move the path for reads and writes to be local (via unmount or LIF migrations). And with NFSv4.x, the statefulness of the connection can be preserved.
That means more time for everyone. Data can be migrated in real time, non-disruptively, based on the storage needs of the client.
For example, I have a volume that lives on node cluster01 of my cDOT cluster:
cluster::> net int show -vserver SVM
(network interface show)
Logical Status Network Current Current Is
Vserver Interface Admin/Oper Address/Mask Node Port Home
----------- ---------- ---------- ------------------ ------------- ------- ----
SVM
data1 up/up 10.63.57.237/18 cluster01 e0c true
data2 up/up 10.63.3.68/18 cluster02 e0c true
2 entries were displayed.
In the above list:
10.63.3.68will be my metadatapath, since that’s where I mounted.
10.63.57.237will be my datapath, as it is local to the physical node cluster02.
When I mount, the TCP connection is established to the node where the data LIF lives:
nfs-client# mount -o minorversion=1 10.63.3.68:/unix /unix
cluster::> network connections active show -remote-ip 10.228.225.140
Vserver Interface Remote
Name Name:Local Port Host:Port Protocol/Service
---------- ---------------------- ---------------------------- ----------------
Node: cluster02
SVM data2:2049 nfs-client.domain.netapp.com:912
TCP/nfs
My metadata path is established to cluster02, but my data volume lives on cluster01.
On a basic cd and ls into the mount, all the traffic is seen on the metadata path. (stuff like GETATTR, ACCESS, etc):
In LAYOUTGET, the client asks the server “where does this filehandle live?”
The server responds with the device ID and physical location of the filehandle.
Then, the client asks “what devices to access that physical data are avaiabe to me?” via GETDEVINFO.
The server responds with the list of available devices/IP addresses.
Once that communication takes place (and note that the conversation occurs in sub-millisecond times), the client then establishes the new TCP connection for reads and writes:
How do I make sure metadata connections don’t pile up?
When you have many clients mounting to an NFS server, you generally want to try to control which nodes those clients are mounting to. In the cloud, this becomes trickier to do, as clients and storage system management may be handled by the cloud providers. So, we’d want to have a noninteractive way to do this.
With ONTAP, you have two options to load balance TCP connections for metadata. You can use the tried and true DNS round-robin method, but the NFS server doesn’t have any idea what IP addresses have been issued by the DNS server, so as a result, there are no guarantees the connections won’t pile up.
Another way to deal with connections is to leverage the ONTAP feature for on-box DNS load balancing. This feature allows storage administrators to set up a DNS forwarding zone on a DNS server (BIND, Active Directory or otherwise) to forward requests to the clustered Data ONTAP data LIFs, which can act as DNS servers complete with SOA records! The cluster will determine which IP address to issue to a client based on the following factors:
CPU load
overall node throughput
This helps ensure that any TCP connection that is established is done so in a logical manner based on performance of the phyical hardware.
What’s great about pNFS is that it is a perfect fit for storage operating systems like ONTAP. NetApp and RedHat worked together closely on the protocol enhancement, and it shows in its overall implementation.
In ONTAP, there is the concept of non-disruptive volume moves. This feature gives storage administrators agility and flexibility in their clusters, as well as enabling service and cloud providers a way to charge based on tiers (pay as you grow!).
For example, if I am a cloud provider, I could have a 24-node cluster as a backend. Some HA pairs could be All-Flash FAS (AFF) nodes for high-performance/low latency workloads. Some HA pairs could be SATA or SAS drives for low performance/high capacity/archive storage. If I am providing storage to a customer that wants to implement high performance computing applications, I could sell them the performance tier. If those applications are only going to run during the summer months, we can use the performance tier, and after the jobs are complete, we can move them back to SATA/SAS drives for storage and even SnapMirror or SnapVault them off to a DR site for safekeeping. Once the job cycle comes back around, I can nondisruptively move the volumes back to flash. That saves the customer money, as they only pay for the performance they’re using, and that saves the cloud provider money since they can free up valuable flash real estate for other customers that need performance tier storage.
What happens when a volume moves in pNFS?
When a volume move occurs, the client is notified of the change via the pNFS calls I mentioned earlier. When the file attempts to OPEN for writing, the server responds, “that file is somewhere else now.”
220 24.971992 10.228.225.140 10.63.3.68 NFS 386 V4 Call (Reply In 221) OPEN DH: 0x76306a29/testfile3
221 24.981737 10.63.3.68 10.228.225.140 NFS 482 V4 Reply (Call In 220) OPEN StateID: 0x1077
If you have an on-premises storage system and want to save storage infrastructure costs by automatically tiering cold data to the cloud or to an on-premises object storage system, you could leverage NetApp FabricPool, which allows you to set tiering policies to chunk off cold blocks of data to more cost effective storage and then retrieve those blocks whenever they are requested by the end user. Again, we’re taking the guesswork and labor out of data management, which is becoming critical in a world driven towards managed services.
As of ONTAP 9.7, NFSv4.1 and pNFS is supported with FlexGroup volumes, which is an intriguing solution.
Part of the challenge of a FlexGroup volume is that you’re guaranteed to have remote I/O across a cluster network when you span multiple nodes. But since pNFS automatically redirects traffic to local paths, you can greatly reduce the amount of intracluster traffic.
A FlexGroup volume operates as a single entity, but is constructed of multiple FlexVol member volumes. Each member volume contains unique files that are not striped across volumes. When NFS operations connect to FlexGroup volumes, ONTAP handles the redirection of operations over a cluster network.
With pNFS, these remote operations are reduced, because the data layout mappings track the member volume locations and local network interfaces; they also redirect reads/writes to the local member volume inside a FlexGroup volume, even though the client only sees a single namespace. This approach enables a scale-out NFS solution that is more seamless and easier to manage, and it also reduces cluster network traffic and balances data network traffic more evenly across nodes.
FlexGroup pNFS differs a bit from FlexVol pNFS. Even though FlexGroup load-balances between metadata servers for file opens, pNFS uses a different algorithm. pNFS tries to direct traffic to the node on which the target file is located. If multiple data interfaces per node are given, connections can be made to each of the LIFs, but only one of the LIFs of the set is used to direct traffic to volumes per network interface.
What workloads should I use with pNFS?
pNFS is leveraging NFSv4.1 and later as its protocol, which means you get all the benefits of NFSv4.1 (security, Kerberos and lock integration, lease-based locks, delegations, ACLs, etc.). But you also get the potential negatives of NFSv4.x, such as higher overhead for operations due to the compound calls, state ID handling, locking, etc. and disruptions during storage failovers that you wouldn’t see with NFSv3 due to the stateful nature of NFSv4.x.
Performance can be severely impacted with some workloads, such as high file count workloads/high metadata workloads (think EDA, software development, etc). Why? Well, recall that pNFS is parallel for reads and writes – but the metadata operations still use a single interface for communication. So if your NFS workload is 80% GETATTR, then 80% of your workload won’t benefit from the localization and load balancing that pNFS provides. Instead, you’ll be using NFSv4.1 as if pNFS were disabled.
Plus, with millions of files, even if you’re doing heavy reads and writes, that means you’re redirecting paths constantly with pNFS (creating millions of DEVICEINFO and LAYOUTGET calls), which may prove more inefficient than simply using NFSv4.1 without pNFS.
pNFS also would need to be supported by the clients you’re using, so if you want to use it for something like VMware datastores, you’re out of luck (for now). VMware currently supports NFSv4.1, but not pNFS (they went with session trunking, which ONTAP does not currently support).
File-based pNFS works best with workloads that do a lot of sequential IO, such as databases, Hadoop/Apache Spark, AI training workloads, or other large file workloads, where reads and writes dominate the IO.
What about the performance?
In TR-4067, I did some basic performance testing on NFSv3 vs. NFSv4.1 for those types of workloads and the results were that pNFS stacked up nicely with NFSv3.
These tests were done using dd in parallel to simulate a sequential I/O workload. This isn’t intended to show the upper limits of the system (I used an AFF 8040 and some VM clients with low RAM and 1GB networks), but instead were intended to show an apples to apples comparison of NFSv3 and NFS4.1 with and without pNFS, using different wsize/rsize values. Be sure to do your own tests before implementing in production.
Note that our completion time for this workload using pNFS was a full 5 minutes faster than NFSv3 using a 1MB wsize/rsize value.
Test (wsize/rsize setting)
Completion Time
NFSv3 (1MB)
15m23s
NFSv3 (256K)
14m17s
NFSv3 (64K)
14m48s
NFSv4.1 (1MB)
15m6s
NFSv4.1 (256K)
12m10s
NFSv4.1 (64K)
15m8s
NFSv4.1 (1MB; pNFS)
10m54s
NFSv4.1 (256K; pNFS)
12m16s
NFSv4.1 (64K; pNFS)
13m57s
NFSv4.1 (1MB; delegations)
13m6s
NFSv4.1 (256K; delegations)
15m25s
NFSv4.1 (64K; delegations)
13m48s
NFSv4.1 (1MB; pNFS + delegations)
11m7s
NFSv4.1 (256K; pNFS + delegations)
13m26s
NFSv4.1 (64K; pNFS + delegations)
10m6s
The IOPS were lower overall for NFSv4.1 than NFSv3; that’s because NFSv4.1 combines operations into single packets. Thus, NFSv4.1 will be less chatty over the network than NFSv3. On the downside, the payloads are larger, so the NFS server has more processing to do for each packet, which can impact CPU, and with more IOPS, you can see a drop in performance due to that overhead.
Where NFSv4.1 beat out NFSv3 was with the latency and throughput – since we can guarantee data locality, we get benefits of fastpathing the reads/writes to the files, rather than the extra processing needed to traverse the cluster network.
Test (wsize/rsize setting)
Average Read Latency (ms)
Average Read Throughput (MB/s)
Average Write Latency (ms)
Average Write Throughput (MB/s)
Average Ops
NFSv3 (1MB)
6
654
27.9
1160
530
NFSv3 (256K)
1.4
766
2.9
1109
2108
NFSv3 (64K)
.2
695
2.2
1110
8791
NFSv4.1 (1MB)
6.5
627
36.8
1400
582
NFSv4.1 (256K)
1.4
712
3.2
1160
2352
NFSv4.1 (64K)
.1
606
1.2
1310
7809
NFSv4.1 (1MB; pNFS)
3.6
840
26.8
1370
818
NFSv4.1 (256K; pNFS)
1.1
807
5.2
1560
2410
NFSv4.1 (64K; pNFS)
.1
835
1.9
1490
8526
NFSv4.1 (1MB; delegations)
5.1
684
32.9
1290
601
NFSv4.1 (256K; delegations)
1.3
648
3.3
1140
1995
NFSv4.1 (64K; delegations)
.1
663
1.3
1000
7822
NFSv4.1 (1MB; pNFS + delegations)
3.8
941
22.4
1110
696
NFSv4.1 (256K; pNFS + delegations)
1.1
795
3.3
1140
2280
NFSv4.1 (64K; pNFS + delegations)
.1
815
1
1170
11130
For high file count workloads, NFSv3 did much better. This test created 800,000 small files (512K) in parallel. For this high metadata workload, NFSv3 completed 2x as fast as NFSv4.1. pNFS added some time savings versus NFSv4.1 without pNFS, but overall, we can see where we may run into problems with this type of workload. Future releases of ONTAP will get better with this type of workload using NFSv4.1 (these tests were on 9.7).
Test (wsize/rsize setting)
Completion Time
CPU %
Average throughput (MB/s)
Average total IOPS
NFSv3 (1MB)
17m29s
32%
351
7696
NFSv3 (256K)
16m34s
34.5%
372
8906
NFSv3 (64K)
16m11s
39%
394
13566
NFSv4.1 (1MB)
38m20s
26%
167
7746
NFSv4.1 (256K)
38m15s
27.5%
167
7957
NFSv4.1 (64K)
38m13s
31%
172
10221
NFSv4.1 pNFS (1MB)
35m44s
27%
171
8330
NFSv4.1 pNFS (256K)
35m9s
28.5%
175
8894
NFSv4.1 pNFS (64K)
36m41s
33%
171
10751
Enter nconnect
One of the keys to pNFS performance is parallelization of operations across volumes, nodes, etc. But it doesn’t necessarily parallelize network connections across these workloads. That’s where the new NFS mount option nconnect comes in.
The purpose of nconnect is to provide multiple transport connections per TCP connection or mount point on a client. This helps increase parallelism and performance for NFS mounts – particularly for single client workloads. Details about nconnect and how it can increase performance for NFS in Cloud Volumes ONTAP can be found in the blog post The Real Baseline Performance Story: NetApp Cloud Volumes Service for AWS. ONTAP 9.8 offers official support for the use of nconnect with NFS mounts, provided the NFS client also supports it. If you would like to use nconnect, check to see if your client version provides it and use ONTAP 9.8 or later. ONTAP 9.8 and later supports nconnect by default with no option needed.
Client support for nconnect varies, but the latest RHEL 8.3 release supports it, as do the latest Ubuntu and SLES releases. Be sure to verify if your OS vendor supports it.
Our Customer Proof of Concept lab (CPOC) did some benchmarking of nconnect with NFSv3 and pNFS using a sequential I/O workload on ONTAP 9.8 and saw some really promising results.
Single NVIDIA DGX-2 client
Ubuntu 20.04.2
NFSv4.1 with pNFS and nconnect
AFF A400 cluster
NetApp FlexGroup volume
256K wsize/rsize
100GbE connections
32 x 1GB files
In these tests, the following throughput results were seen. Latency for both were sub 1ms.
Test
Bandwidth
NFSv3
10.2 GB/s
NFSv4.1/pNFS
21.9 GB/s
Both NFSv3 and NFSv4.1 used nconnect=16
In these tests, NFSv4.1 with pNFS doubled the performance for the sequential read workload at 250us latency. Since the files were 1GB in size, the reads were almost entirely from the controller RAM, but it’s not unreasonable to see that as the reality for a majority of workloads, as most systems have enough RAM to see similar results.
David Arnette and I discuss it a bit in this podcast:
Note: Benchmark tests such as SAS iotest will purposely recommend setting file sizes larger than the system RAM to avoid any caching benefits and instead will measure the network bandwidth of the transfer. In real world application scenarios, RAM, network, storage and CPU are all working together to create the best possible performance scenarios.
pNFS Best Practices with ONTAP
pNFS best practices in ONTAP don’t differ much from normal NAS best practices, but here are a few to keep in mind. In general:
Use the latest supported client OS version.
Use the latest supported ONTAP patch release.
Create a data LIF per node, per SVM to ensure data locality for all nodes.
Avoid using LIF migration on the metadata server data LIF, because NFSv4.1 is a stateful protocol and LIF migrations can cause brief outages as the NFS states are reestablished.
In environments with multiple NFSv4.1 clients mounting, balance the metadata server connections across multiple nodes to avoid piling up metadata operations on a single node or network interface.
If possible, avoid using multiple data LIFs on the same node in an SVM.
In general, avoid mounting NFSv3 and NFSv4.x on the same datasets. If you can’t avoid this, check with the application vendor to ensure that locking can be managed properly.
If you’re using NFS referrals with pNFS, keep in mind that referrals establish a local metadata server, but data I/O still redirect. With FlexGroup volumes, the member volumes might live on multiple nodes, so NFS referrals aren’t of much use. Instead, use DNS load balancing to spread out connections.
We’re pretty close to seeing it available for general availability, so in preparation for that, here are some resources available for you to figure out exactly what is in the latest release.
Insight Presentation: Do More with Less in the Latest Release of NetApp ONTAP
This is the full video presentation of the ONTAP 9.8 session from NetApp Insight.
This new TR is basically the release notes on steroids; we cover the new features and give some explanations and primers to them. We’re using the new GitHub based documentation method for this:
Welcome to the Episode 263, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week, NetApp ONTAP Virtualization PM Karl Konnerth (@konnerth) and Virtualization TME Chance Bingen (@CB8MyDataCenter) join me to discuss the latest updates to ONTAP for VMware virtualization integration, as well as what sessions they recommend for NetApp Insight 2020.
For the VMware in ONTAP Best Practices TR, see here:
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Or, click the “view transcript” button:
Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Tech ONTAP Community
We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.
Welcome to the Episode 261, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week, I was lucky enough to snag a long-time NetApp customer and engineering partner DreamWorks Animation to talk about how the studio leverages NetApp ONTAP to build some of the most creative and effects intensive CG animated films on the planet.
Podcast Transcriptions
We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Or, click the “view transcript” button:
Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Welcome to the Episode 228, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, we discuss FlexPod and the new initiative to create validated designs for industry verticals. First up – Healthcare and Epic software with NetApp Sr. Product Manager for Converged Infrastructure, Ketan Mota (ketan.mota@netapp.com) and NetApp Solutions Architect for Healthcare, Brian O’Mahony (omahony@netapp.com).
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Finding the Podcast
You can find this week’s episode here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
Welcome to the Episode 227, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, Adam Knight (@damknight) of Pacific Biosciences joins us to discuss how PacBio uses ONTAP for all of its unstructured NAS workload requirements, with a focus on FlexGroup volumes!
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Finding the Podcast
You can find this week’s episode here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
Thoughts on Industrial Computing. Or sometimes economics or music or trail runnning or carpentry. Employee of Datto. Proud New Englander and Schaghticoke. Site is my own.
