Welcome to the Episode 228, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, we discuss FlexPod and the new initiative to create validated designs for industry verticals. First up – Healthcare and Epic software with NetApp Sr. Product Manager for Converged Infrastructure, Ketan Mota (ketan.mota@netapp.com) and NetApp Solutions Architect for Healthcare, Brian O’Mahony (omahony@netapp.com).
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Finding the Podcast
You can find this week’s episode here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
Welcome to the Episode 227, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, Adam Knight (@damknight) of Pacific Biosciences joins us to discuss how PacBio uses ONTAP for all of its unstructured NAS workload requirements, with a focus on FlexGroup volumes!
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Finding the Podcast
You can find this week’s episode here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
The TRs cover those features, and there are some updates to other areas that might not have been as clear as they could have been. I also added some new use cases.
Also, check out the newest FlexGroup episode of the Tech ONTAP Podcast:
Welcome to the Episode 219, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, we invite the NetApp FlexGroup Technical Director, Dan Tennant, and FlexGroup developer Jessica Peters, to talk to us about the ins and outs of converting a FlexVol to a FlexGroup in-place, with no copy and no outage!
I also cover the process in detail in this blog post:
Just use the search field to look for words you want to read more about. (For example, search for “storage”)
Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!
Finding the Podcast
You can find this week’s episode here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
UPDATED: Added 500 million file test to this to show that file count doesn’t matter. 🙂
NetApp announced ONTAP 9.7 at Insight 2019 in Las Vegas, which included a number of new features. But mainly, ONTAP 9.7 focuses on making storage management in ONTAP simpler.
One of the new features that will help make things easier is the new FlexGroup conversion feature, which allows in-place conversion of a FlexVol to a FlexGroup volume without the need to do a file copy.
Best of all, this conversion takes a matter of seconds without needing to remount clients!
I know it sounds too good to be true, but what would you rather do: spend days copying terabytes of data over the network, or run a single command that converts the volume in place without touching the data?
As you can imagine, a lot of people are pretty stoked about being able to convert volumes without copying data, so I wanted to write up something to point people to as the questions inevitably start rolling in. This blog will cover how it works and what caveats there are. The blog will be a bit long, but I wanted to cover all the bases. Look for this information to be included in TR-4571 soon, as well as a new FlexGroup conversion podcast in the coming weeks.
Why would I want to convert a volume to a FlexGroup?
FlexGroup volumes offer a few advantages over FlexVol volume, such as:
Ability to expand beyond 100TB and 2 billion files in a single volume
Ability to scale out capacity or performance non-disruptively
Multi-threaded performance for high ingest workloads
Simplification of volume management and deployment
For example, perhaps you have a workload that is growing rapidly and you don’t want to have to migrate the data, but still want to provide more capacity. Or perhaps a workload’s performance just isn’t cutting it on a FlexVol, so you want to provide better performance handling with a FlexGroup. Converting can help here.
When would I not want to convert a FlexVol?
Converting a FlexVol to a FlexGroup might not always be the best option. If you have features you require in FlexVol that aren’t available in FlexGroup volumes, then you should hold off. For example, SVM-DR and cascading SnapMirrors aren’t supported in ONTAP 9.7, so if you need those, you should stay with FlexVols.
Also, if you have a FlexVol that’s already very large (80-100TB) and already very full (80-90%) then you might want to copy the data rather than convert, as the converted FlexGroup volume would then have a very large, very full member volume, which could create performance issues and doesn’t really fully resolve your capacity issues – particularly if that dataset contains files that grow over time.
For example, if you have a FlexVol that is 100TB in capacity and 90TB used, it would look like this:
If you were to convert this 90% full volume to a FlexGroup, then you’d have a 90% full member volume. Once you add new member volumes, they’d be 100TB each and 0% full, meaning they’d take on a majority of new workloads. The data would not rebalance and if the original files grew over time, you could still run out of space with nowhere to go (since 100TB is the maximum member volume size).
Things that would block a conversion
ONTAP will block conversion of a FlexVol for the following reasons:
The ONTAP version isn’t 9.7 on all nodes
ONTAP upgrade issues preventing conversion
A FlexVol volume was transitioned from 7-Mode using 7MTT
Something is enabled on the volume that isn’t supported with FlexGroups yet (SAN LUNs, Windows NFS, SMB1, part of a fan-out/cascade snapmirror, SVM-DR, Snapshot naming/autodelete, vmalign set, SnapLock, space SLO, logical space enforcement/reporting, etc.)
FlexClones are present (The volume being converted can’t be a parent nor a clone)
The volume is a FlexCache origin volume
Snapshots with snap-ids greater than 255
Storage efficiencies are enabled (can be re-enabled after)
The volume is a source of a snapmirror and the destination has not been converted yet
The volume is part of an active (not quiesced) snapmirror
Quotas enabled (must be disabled first, then re-enabled after)
To convert a FlexVol volume to a FlexGroup volume in ONTAP 9.7, you run a single, simple command in advanced privilege:
cluster::*> volume conversion start ?
-vserver <vserver name> *Vserver Name
[-volume] <volume name> *Volume Name
[ -check-only [true] ] *Validate the Conversion Only
[ -foreground [true] ] *Foreground Process (default: true)
When you run this command, it will take a single FlexVol and convert it into a FlexGroup volume with one member. You can even run a validation of the conversion before you do the real thing!
The process is 1:1, so you can’t currently convert multiple FlexVols into a single FlexGroup. Once the conversion is done, you will have a single member FlexGroup volume, which you can then add more member volumes of the same size to increase capacity and performance.
Other considerations/caveats
While the actual conversion process is simple, there are some considerations to think of before converting. Most of these considerations will go away in each ONTAP release as support is added for features, but it’s still prudent to call them out here.
Once the initial conversion is done, ONTAP will unmount the volume internally and remount it to get the new FlexGroup information into the appropriate places. Clients won’t have to remount/reconnect, but will see a disruption that last less than 1 minute while this takes place. Data doesn’t change at all – filehandles all stay the same.
FabricPool doesn’t need anything. It just works. No need to rehydrate data on-prem.
Snapshot copies will remain and available for clients to access data from, but you won’t be able to restore the volume using them via snaprestore commands. Those snapshots get marked as “pre-conversion.”
SnapMirrors will pick up where they left off without needing to rebaseline, provided the source and destination volumes have both been converted. But no snapmirror restores of the volume; just file retrieval from clients. Snapmirror destinations need to be converted first.
FlexClones will need to be deleted or split from the volume to be converted.
Storage efficiencies will need to be disabled during the conversion, but your space savings will be preserved after the convert
FlexCache instances with an origin volume being converted will need to be deleted.
Space guarantees can impact how large a FlexGroup volume can get if they’re set to volume guarantee. New member volumes will need to be the same size as the existing members, so you’d need adequate space to honor those.
Quotas are supported in FlexGroup volumes, but are done a bit differently than in FlexVol volumes. So, while the convert is being done, quotas have to be disabled (quota off) and then re-enabled later (quota on).
Also, conversion to FlexGroup volumes is a one way street after you expand it, so be sure you’re ready to make the jump. If anything goes wrong during the conversion process, there is a “rescue” method that support can help you use to get out of the pickle, so your data will be safe.
When you expand the FlexGroup to add new member volumes, they will be the same size as the converted member volume, so be sure there is adequate space available. Additionally, the existing data that resides in the original volume will remain in that member volume. Data does not re-distribute. Instead, the FlexGroup will favor newly added member volumes for new files.
Nervous about convert?
Well, ONTAP has features for that.
If you don’t feel comfortable about converting your production FlexVol to a FlexGroup right away, you have options.
First of all, remember that we have the ability to run a check on the convert command with -check-only true. That tells us what pre-requisites we might be missing.
cluster::*> volume conversion start -vserver DEMO -volume flexvol -foreground true -check-only true
Error: command failed: Cannot convert volume "flexvol" in Vserver "DEMO" to a FlexGroup. Correct the following issues and retry the command:
* The volume has Snapshot copies with IDs greater than 255. Use the (privilege: advanced) "volume snapshot show -vserver DEMO -volume flexvol -logical-snap-id >255 -fields logical-snap-id" command to list the Snapshot copies
with IDs greater than 255 then delete them using the "snapshot delete -vserver DEMO -volume flexvol" command.
* Quotas are enabled. Use the 'volume quota off -vserver DEMO -volume flexvol' command to disable quotas.
* Cannot convert because the source "flexvol" of a SnapMirror relationship is source to more than one SnapMirror relationship. Delete other Snapmirror relationships, and then try the conversion of the source "flexvol" volume.
* Only volumes with logical space reporting disabled can be converted. Use the 'volume modify -vserver DEMO -volume flexvol -is-space-reporting-logical false' command to disable logical space reporting.
Also, remember, ONTAP has the ability to create multiple storage virtual machines, which can be fenced off from network access. This can be used to test things, such as volume conversion. The only trick is getting a copy of that data over… but it’s really not that tricky.
Option 1: SnapMirror
You can create a SnapMirror of your “to be converted” volume to the same SVM or a new SVM. Then, break the mirror and delete the relationship. Now you have a sandbox copy of your volume, complete with snapshots, to test out conversion, expansion and performance.
Option 2: FlexClone/volume rehost
If you don’t have SnapMirror or want to try a method that is less taxing on your network, you can use a combination of FlexClone (instant copy of your volume backed by a snapshot) and volume rehost (instant move of the volume from one SVM to another). Keep in mind that FlexClones themselves can’t be rehosted, but you can split the clone and then rehost.
Essentially, the process is:
FlexClone create
FlexClone split
Volume rehost to new SVM (or convert on the existing SVM)
Profit!
Sample conversion
Before I converted a volume, I added around 300,000 files to help determine how long the process might take with a lot of files present.
cluster::*> df -i lotsafiles
Filesystem iused ifree %iused Mounted on Vserver
/vol/lotsafiles/ 330197 20920929 1% /lotsafiles DEMO
cluster::*> volume show lots*
Vserver Volume Aggregate State Type Size Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
DEMO lotsafiles aggr1_node1 online RW 10TB 7.33TB 0%
First, let’s try out the validation:
cluster::*> volume conversion start -vserver DEMO -volume lotsafiles -foreground true -check-only true
Error: command failed: Cannot convert volume "lotsafiles" in Vserver "DEMO" to a FlexGroup. Correct the following issues and retry the command:
* SMB1 is enabled on Vserver "DEMO". Use the 'vserver cifs options modify -smb1-enabled false -vserver DEMO' command to disable SMB1.
* The volume contains LUNs. Use the "lun delete -vserver DEMO -volume lotsafiles -lun *" command to remove the LUNs, or use the "lun move start" command to relocate the LUNs to other
FlexVols.
* NFSv3 MS-DOS client support is enabled on Vserver "DEMO". Use the "vserver nfs modify -vserver DEMO -v3-ms-dos-client disabled" command to disable NFSv3 MS-DOS client support on the
Vserver. Note that disabling this support will disable access for all NFSv3 MS-DOS clients connected to Vserver "DEMO".
As you can see, we have some blockers, such as SMB1 and the LUN I created (to intentionally break conversion). So, I clear them with the recommendations and run it again and see some of our caveats:
cluster::*> volume conversion start -vserver DEMO -volume lotsafiles -foreground true -check-only true
Conversion of volume "lotsafiles" in Vserver "DEMO" to a FlexGroup can proceed with the following warnings:
* After the volume is converted to a FlexGroup, it will not be possible to change it back to a flexible volume.
* Converting flexible volume "lotsafiles" in Vserver "DEMO" to a FlexGroup will cause the state of all Snapshot copies from the volume to be set to "pre-conversion". Pre-conversion Snapshot copies cannot be restored.
Now, let’s convert. But, first, I’ll start a script that takes a while to complete, while also monitoring performance during the conversion using Active IQ Performance Manager.
The conversion of the volume took less than 1 minute, and the only disruption I saw was a slight drop in IOPS:
cluster::*> volume conversion start -vserver DEMO -volume lotsafiles -foreground true
Warning: After the volume is converted to a FlexGroup, it will not be possible to change it back to a flexible volume.
Do you want to continue? {y|n}: y
Warning: Converting flexible volume "lotsafiles" in Vserver "DEMO" to a FlexGroup will cause the state of all Snapshot copies from the volume to be set to "pre-conversion". Pre-conversion Snapshot
copies cannot be restored.
Do you want to continue? {y|n}: y
[Job 23671] Job succeeded: success
Now, I can add more member volumes using “volume expand”:
cluster::*> volume expand -vserver DEMO -volume lotsafiles -aggr-list aggr1_node1,aggr1_node2 -aggr-list-multiplier 2
Warning: The following number of constituents of size 10TB will be added to FlexGroup "lotsafiles": 4. Expanding the FlexGroup will cause the state of all Snapshot copies to be set to "partial".
Partial Snapshot copies cannot be restored.
Do you want to continue? {y|n}: y
Warning: FlexGroup "lotsafiles" is a converted flexible volume. If this volume is expanded, it will no longer be able to be converted back to being a flexible volume.
Do you want to continue? {y|n}: y
[Job 23676] Job succeeded: Successful
But remember, the data doesn’t redistribute. The original member volume will keep the files in place:
Now, let’s take a look at a volume that is in a SnapMirror.
cluster::*> snapmirror show -destination-path data_dst -fields state
source-path destination-path state
----------- ---------------- ------------
DEMO:data DEMO:data_dst Snapmirrored
If I try to convert the source, I get an error:
cluster::*> vol conversion start -vserver DEMO -volume data -check-only true
Error: command failed: Cannot convert volume "data" in Vserver "DEMO" to a FlexGroup. Correct the following issues and retry the command:
* Cannot convert source volume "data" because destination volume "data_dst" of the SnapMirror relationship with "data" as the source is not converted. First check if the source can be converted to a FlexGroup volume using "vol
conversion start -volume data -convert-to flexgroup -check-only true". If the conversion of the source can proceed then first convert the destination and then convert the source.
So, I’d need to convert the destination first. To do that, I need to quiesce the snapmirror:
cluster::*> vol conversion start -vserver DEMO -volume data_dst -check-only true
Error: command failed: Cannot convert volume "data_dst" in Vserver "DEMO" to a FlexGroup. Correct the following issues and retry the command:
* The relationship was not quiesced. Quiesce SnapMirror relationship using "snapmirror quiesce -destination-path data_dst" and then try the conversion.
Here we go…
cluster::*> snapmirror quiesce -destination-path DEMO:data_dst
Operation succeeded: snapmirror quiesce for destination "DEMO:data_dst".
cluster::*> vol conversion start -vserver DEMO -volume data_dst -check-only true
Conversion of volume "data_dst" in Vserver "DEMO" to a FlexGroup can proceed with the following warnings:
* After the volume is converted to a FlexGroup, it will not be possible to change it back to a flexible volume.
* Converting flexible volume "data_dst" in Vserver "DEMO" to a FlexGroup will cause the state of all Snapshot copies from the volume to be set to "pre-conversion". Pre-conversion Snapshot copies cannot be restored.
When I convert the volume, it lets me know my next steps:
cluster::*> vol conversion start -vserver DEMO -volume data_dst
Warning: After the volume is converted to a FlexGroup, it will not be possible to change it back to a flexible volume.
Do you want to continue? {y|n}: y
Warning: Converting flexible volume "data_dst" in Vserver "DEMO" to a FlexGroup will cause the state of all Snapshot copies from the volume to be set to "pre-conversion". Pre-conversion Snapshot copies cannot be restored.
Do you want to continue? {y|n}: y
[Job 23710] Job succeeded: SnapMirror destination volume "data_dst" has been successfully converted to a FlexGroup volume. You must now convert the relationship's source volume, "DEMO:data", to a FlexGroup. Then, re-establish the SnapMirror relationship using the "snapmirror resync" command.
Now I convert the source volume…
cluster::*> vol conversion start -vserver DEMO -volume data
Warning: After the volume is converted to a FlexGroup, it will not be possible to change it back to a flexible volume.
Do you want to continue? {y|n}: y
Warning: Converting flexible volume "data" in Vserver "DEMO" to a FlexGroup will cause the state of all Snapshot copies from the volume to be set to "pre-conversion". Pre-conversion Snapshot copies cannot be restored.
Do you want to continue? {y|n}: y
[Job 23712] Job succeeded: success
And resync the mirror:
cluster::*> snapmirror resync -destination-path DEMO:data_dst
Operation is queued: snapmirror resync to destination "DEMO:data_dst".
cluster::*> snapmirror show -destination-path DEMO:data_dst -fields state
source-path destination-path state
----------- ---------------- ------------
DEMO:data DEMO:data_dst Snapmirrored
While that’s fine and all, the most important part of a snapmirror is the restore. So let’s see if I can access files from the destination volume’s snapshot.
First, I mount the source and destination and compare ls output:
# mount -o nfsvers=3 DEMO:/data_dst /dst
# mount -o nfsvers=3 DEMO:/data /data
# ls -lah /data
total 14G
drwxrwxrwx 6 root root 4.0K Nov 14 11:57 .
dr-xr-xr-x. 54 root root 4.0K Nov 15 10:08 ..
drwxrwxrwx 2 root root 4.0K Sep 14 2018 cifslink
drwxr-xr-x 12 root root 4.0K Nov 16 2018 nas
-rwxrwxrwx 1 prof1 ProfGroup 0 Oct 3 14:32 newfile
drwxrwxrwx 5 root root 4.0K Nov 15 10:06 .snapshot
lrwxrwxrwx 1 root root 23 Sep 14 2018 symlink -> /shared/unix/linkedfile
drwxrwxrwx 2 root bin 4.0K Jan 31 2019 test
drwxrwxrwx 3 root root 4.0K Sep 14 2018 unix
-rwxrwxrwx 1 newuser1 ProfGroup 0 Jan 14 2019 userfile
-rwxrwxrwx 1 root root 6.7G Nov 14 11:58 Windows2.iso
-rwxrwxrwx 1 root root 6.7G Nov 14 11:37 Windows.iso
# ls -lah /dst
total 14G
drwxrwxrwx 6 root root 4.0K Nov 14 11:57 .
dr-xr-xr-x. 54 root root 4.0K Nov 15 10:08 ..
drwxrwxrwx 2 root root 4.0K Sep 14 2018 cifslink
dr-xr-xr-x 2 root root 0 Nov 15 2018 nas
-rwxrwxrwx 1 prof1 ProfGroup 0 Oct 3 14:32 newfile
drwxrwxrwx 4 root root 4.0K Nov 15 10:05 .snapshot
lrwxrwxrwx 1 root root 23 Sep 14 2018 symlink -> /shared/unix/linkedfile
drwxrwxrwx 2 root bin 4.0K Jan 31 2019 test
drwxrwxrwx 3 root root 4.0K Sep 14 2018 unix
-rwxrwxrwx 1 newuser1 ProfGroup 0 Jan 14 2019 userfile
-rwxrwxrwx 1 root root 6.7G Nov 14 11:58 Windows2.iso
-rwxrwxrwx 1 root root 6.7G Nov 14 11:37 Windows.iso
And if I ls to the snapshot in the destination volume…
# ls -lah /dst/.snapshot/snapmirror.7e3cc08e-d9b3-11e6-85e2-00a0986b1210_2163227795.2019-11-15_100555/
total 14G
drwxrwxrwx 6 root root 4.0K Nov 14 11:57 .
drwxrwxrwx 4 root root 4.0K Nov 15 10:05 ..
drwxrwxrwx 2 root root 4.0K Sep 14 2018 cifslink
dr-xr-xr-x 2 root root 0 Nov 15 2018 nas
-rwxrwxrwx 1 prof1 ProfGroup 0 Oct 3 14:32 newfile
lrwxrwxrwx 1 root root 23 Sep 14 2018 symlink -> /shared/unix/linkedfile
drwxrwxrwx 2 root bin 4.0K Jan 31 2019 test
drwxrwxrwx 3 root root 4.0K Sep 14 2018 unix
-rwxrwxrwx 1 newuser1 ProfGroup 0 Jan 14 2019 userfile
-rwxrwxrwx 1 root root 6.7G Nov 14 11:58 Windows2.iso
-rwxrwxrwx 1 root root 6.7G Nov 14 11:37 Windows.iso
Everything is there!
Now, I expand the FlexGroup source to give us more capacity:
cluster::*> volume expand -vserver DEMO -volume data -aggr-list aggr1_node1,aggr1_node2 -aggr-list-multiplier
Warning: The following number of constituents of size 30TB will be added to FlexGroup "data": 4. Expanding the FlexGroup will cause the state of all Snapshot copies to be set to "partial". Partial Snapshot copies cannot be restored.
Do you want to continue? {y|n}: y
[Job 23720] Job succeeded: Successful
If you notice, my source volume now has 5 member volumes. My destination volume… only has one:
No worries! Just update the mirror and ONTAP will fix it for you.
cluster::*> snapmirror update -destination-path DEMO:data_dst
Operation is queued: snapmirror update of destination "DEMO:data_dst".
The update will initially fail with the following:
Last Transfer Error: A SnapMirror transfer for the relationship with destination FlexGroup "DEMO:data_dst" was aborted because the source FlexGroup was expanded. A SnapMirror AutoExpand job with id "23727" was created to expand the destination FlexGroup and to trigger a SnapMirror transfer for the SnapMirror relationship. After the SnapMirror transfer is successful, the "healthy" field of the SnapMirror relationship will be set to "true". The job can be monitored using either the "job show -id 23727" or "job history show -id 23727" commands.
The job will expand the volume and then we can update again:
cluster::*> job show -id 23727
Owning
Job ID Name Vserver Node State
------ -------------------- ---------- -------------- ----------
23727 Snapmirror Expand cluster
node1
Success
Description: SnapMirror FG Expand data_dst
cluster::*> snapmirror show -destination-path DEMO:data_dst -fields state
source-path destination-path state
----------- ---------------- ------------
DEMO:data DEMO:data_dst Snapmirrored
Now both FlexGroup volumes have the same number of members:
So, there you have it… a quick and easy way to move from FlexVol volumes to FlexGroups!
Addendum: Does a High File Count Impact the Convert Process?
Short answer: No!
In a comment a few weeks ago, someone pointed out, rightly, that my 300k file volume wasn’t a *true* high file count. So I set out to create 500 million files and convert that volume. The hardest part was creating 500 million files, but I finally got there:
Since it took me so long to create that many files, I went ahead and created a FlexClone volume of it and split it, so I could keep the origin volume intact, because who doesn’t need 500 million files laying around?
Fun fact: That process *did* take a while – about 30 minutes:
cluster::*> vol clone split start -vserver DEMO -flexclone fvconvert -foreground true
Warning: Are you sure you want to split clone volume fvconvert in Vserver DEMO ? {y|n}: y
[Job 24230] 0% inodes processed.
cluster::*> job history show -id 24230 -fields starttime,endtime
node record vserver endtime starttime
------------------ ------- --------------- -------------- --------------
node1 2832338 cluster 12/09 10:27:08 12/09 09:58:16
After the clone split, I ran the check to see if we were good to go. I did have to run a “volume clone sharing-by-split undo” to get rid of shared FlexClone blocks which took a while, but after that:
cluster::*> volume conversion start -vserver DEMO -volume fvconvert -foreground true -check-only true
Conversion of volume "fvconvert" in Vserver "DEMO" to a FlexGroup can proceed with the following warnings:
* After the volume is converted to a FlexGroup, it will not be possible to change it back to a flexible volume.
I went ahead and ran the same script I was running earlier to generate load and watched the statistics on the cluster to see if we hit any outage. Again, the convert took seconds (with 500 million files) and there was just a small blip.
cluster::*> volume conversion start -vserver DEMO -volume fvconvert -foreground true
Warning: After the volume is converted to a FlexGroup, it will not be possible to change it back to a flexible volume.
Do you want to continue? {y|n}: y
[Job 24259] Job succeeded: success
Then, as the job was running I added new member volumes to the FlexGroup volume – again, no disruption.
cluster::*> volume expand -vserver DEMO -volume fvconvert -aggr-list aggr1_node1 -aggr-list-multiplier 3 -foreground true
Info: Unable to get information for Snapshot copies of volume "fvconvert" on Vserver "DEMO". Reason: No such snapshot.
Warning: The following number of constituents of size 40TB will be added to FlexGroup "fvconvert": 3.
Do you want to continue? {y|n}: y
[Job 24261] Job succeeded: Successful
Then 4 more member volumes:
cluster::*> volume expand -vserver DEMO -volume fvconvert -aggr-list aggr1_node2 -aggr-list-multiplier 4
Info: Unable to get information for Snapshot copies of volume "fvconvert" on Vserver "DEMO". Reason: No such snapshot.
Warning: The following number of constituents of size 40TB will be added to FlexGroup "fvconvert": 4.
Do you want to continue? {y|n}: y
[Job 24264] Job succeeded: Successful
Plus, I started to see more IOPs for the workload, and the job itself took much less time overall than when I ran it on a FlexVol.
For a video of the capture, check it out here:
This was the job on the FlexVol:
# python file-create.py /fvconvert/files
Starting overall work: 2019-12-09 10:32:21.966337
End overall work: 2019-12-09 12:11:15.990707
total time: 5934.024611
This is how long it took on the FlexVol converted to a FlexGroup (with added member volumes):
# python file-create.py /fvconvert/files2
Starting overall work: 2019-12-10 11:02:28.621532
End overall work: 2019-12-10 12:22:48.523772
total time: 4819.95753193
And, for fun, I kicked it off again on the new FlexGroup. This time, I wanted to see how much faster the job ran, as well as how the files distributed on the more empty FlexVol members.
Remember, we started out with the newer member volumes all at less than 1% of files used (3.3 million of 2 billion possible files). The member volume that was converted from a FlexVol was using 25% of the total files (500 million of 2 billion).
After the job ran, we saw a ~3.2 million file count delta on the original member volume and a ~3.58 million file count delta on all the other members, which means we’re still balancing across all member volumes, but favoring the less full ones.
With the new FlexGroup, converted from a FlexVol, our job time dropped from 5900 seconds to 4656 seconds and we were able to push 2x the amount of IOPs:
# python file-create.py /fvconvert/files3
Starting overall work: 2019-12-10 13:14:26.816860
End overall work: 2019-12-10 14:32:03.565705
total time: 4656.76723099
As you can see, there’s an imbalance of files and data in these member volumes (way more in the original FlexVol), but performance still blows away the previous FlexVol performance because we are doing more efficient work across multiple nodes.
Welcome to the Episode 209, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, we talk genomics and how to design an end to end solution using NetApp products across the portfolio with Cloud Solution Architect Florian Fledhaus (@flfeld).
Finding the Podcast
You can find this week’s episode here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
But what if i told you that you could use XCP to delete millions of files 5-6x faster than running rm on an NFS client?
Wait… why would I delete millions of files?
Normally, you wouldn’t. But in some workflows, such scratch space, this is what happens. A bunch of small files get generated and then deleted once the work is done.
I ran a simple test in my lab where I had a flexgroup volume with ~37 million files in it.
I took a snapshot of that data so I could restore it later for XCP to delete and then ran rm -rf on it from a client. It took 20 hours:
# time rm -rf /flexgroup/*
real 1213m4.652s
user 1m39.703s
sys 41m16.978s
Then I restored the snapshot and deleted the same ~37 million files using XCP. That took roughly 3.5 hours:
# time xcp diag -rmrf 10.193.67.219:/flexgroup_16
real 218m17.765s
user 149m16.132s
sys 40m47.427s
So, if you have a workflow that requires you to delete large amounts of data that normally takes you FOREVER, try XCP next time…
These are VMs with limited RAM and 1GB network connections, so I’d imagine with bigger, beefier servers, those times could come down a bit more. But in an apples to apples test, XCP wins again!
There are some new features in ONTAP 9.6 for FlexGroup volumes, including:
Elastic Sizing
MetroCluster support
SMB CA shares
FlexGroup rename/shrink
The TRs cover those features, and there are some updates to other areas that might not have been as clear as they could have been. I also added some new use cases.
Also, check out the newest FlexGroup episode of the Tech ONTAP Podcast:
TR Update List
Here’s the list of FlexGroup TRs that have been updated for ONTAP 9.6:
Spring has barely sprung and already we’re seeing the latest ONTAP release!
ONTAP 9.6 has been announced and it’s chock full of goodness.
For the short overview podcast, check this link out:
For the longer overview, go here:
This time, the prevailing themes are simplicity, security, data protection and storage efficiency. We’ll have a few podcasts centered around some of the new features, as well. They will be available to download in a few weeks, but for now, let’s discuss what’s in it.
What’s new?
Generally speaking, new stuff in ONTAP comes in the following forms:
New features
Enhanced features
With the 6 month cadence, features are often phased in, with new features being released with stability as the top priority. Feature parity comes in chunks in later releases. Bug fixes are a part of every ONTAP release.
In ONTAP 9.6, there’s actually a new change that doesn’t really fall into the three categories above. Instead, it’s a departure from how ONTAP handled support for releases.
Long Term vs. Short Term
Around ONTAP 9.0, ONTAP adopted the 6 month cadence for releases. A decision was made to make odd numbered/Fall releases “long term” support releases (LTS) and even numbered/Spring releases “short term” support releases (STS). Short term releases would get a shorter runway of official software support than long term releases.
There was an unintended consequence, however – fewer people wanted to use short term releases, even though the short term releases had the same rigorous quality testing and stability as long term releases. “Short term” had negative connotations, and customers didn’t want to face the prospect of being forced to upgrade sooner, even if that upgrade cycle was years after the release.
So, ONTAP 9.6 scraps the short term release. All new ONTAP releases will be considered long term support releases and will have:
3 years full support (technical support, root cause analysis, SW and documentation available online, frequent service updates in year 1)
2 years limited support (technical support, root cause analysis, SW and documentation available online, no service updates)
3 years of self-service support (documentation available online, no technical support from NetApp, no service updates)
New Features
ONTAP 9.6 focuses on upping the game on simplicity in order to help consumers of ONTAP – new and old – provision storage and move in and out of the cloud much easier.
Most of the stuff in ONTAP 9.6 centers around feature enhancements, but there is one main new feature that I think merits discussion…
REST APIs
I cover REST APIs in another blog post, but essentially, ONTAP is adding support for a standard interface that can query or make changes to an ONTAP cluster via normal HTTP GET and PUT requests.
This allows storage administrators to more easily automate routine tasks without needing to use a proprietary model like ZAPI or ONTAP-specific PowerShell cmdlets. REST APIs also offer a performance improvement over ZAPI calls.
Aggregate level encryption
Another new feature in ONTAP 9.6 centers around the ability to encrypt data at rest at the aggregate level. Previously, you could either encrypt data at rest using drive-enabled hardare encryption on NetApp Storage Encryption (NSE) systems or software-based volume level encryption.
With NSE drives, you can encrypt an entire HA pair, but that means everything on that HA pair is encrypted. That provides less flexibility in management and limits your options with regards to which drives you can use in a system. NSE also means a common key for all volumes on the cluster, and thus no separation for tenants on different SVMs.
NetApp Volume Encryption (NVE) gives more granularity in controlling secure data access, but because each volume has its own encryption key, you lose some simplicity (having to generate a key for every new volume) and storage efficiency (unique keys for each volume means aggregate level deduplication doesn’t work with NVE).
Aggregate level encryption provides simplicity by way of setting an encryption once when the aggregate is created and all volumes in the aggregate sharing the same encryption key, and it allows common blocks across all volumes in the aggregate to be deduplicated.
Feature Enhancements – Simplicity and Productivity
This is where most of the ONTAP 9.6 update does its magic – feature enhancements. One of the prevailing enhancements in ONTAP 9.6 is the focus on simplicity and productivity.
System Manager Enhancements – System Manager is getting a bit of an overhaul. For starters, it’s no longer called OnCommand – instead, it has been re-branded as ONTAP System Manager.
ONTAP System Manager also features:
Redesigned page views and simpler workflows optimized for use by IT generalists
Visualizations of network topology and storage hardware
Intelligent capacity reporting of on-premises and cloud usage at a glance
Preview “view” version only in ONTAP 9.6 – feedback welcomed, write to: voc@netapp.com
The preview version can be viewed by clicking on “Try the new experience” in the top left corner:
There are also some improvements added with the new System Manager view.
For capacity, you see combined capacity and efficiency views as soon as you open ONTAP System Manager.
For example:
Performance views in System Manager are getting an update as well. Previously, if you wanted to see system performance in System Manager, you only were given a real-time view of performance that did not capture historical data.
In the new view, we store up to a year of perf data in System Manager for historical views that can help narrow down hard-to-pinpoint performance issues in your ONTAP cluster.
ONTAP System Manager has also streamlined how protocols are managed. ONTAP is one of the only storage systems in the industry that can support SAN and NAS protocols, but managing them in previous System Manager iterations required more clicks.
In the new System Manager view, we have a single splash page:
There’s also a network diagram in the dashboard that gives a breakdown of available ports, LIFs and SVMs.
Expect more to come for System Manager enhancements in future ONTAP releases!
Simplicity doesn’t just mean “make System Manager better,” however. There are a number of other feature enhancements that improve ONTAP simplicity, as well as overall productivity.
FabricPool simplicity – Automatic inactive data reporting and new tiering policy
In ONTAP 9.4, a feature enhancement was added where you could have ONTAP analyze your cold data and report back how much space savings you’d get from enabling FabricPool. In ONTAP 9.6, this functionality is now on by default.
Additionally, a new tiering policy has been added called “all,” which will allow storage administrators to easily tier all data from a volume instantly, without having to wait for the cold data policy to kick in. This comes with some usage guidelines:
Use “all” policy only when importing already cold data
Or data that will be read sequentially only
Don’t use the “all” policy with random read or random read/write workloads
Random reads from the cloud tier are slow, and
Write updates (overwrites) fragment object storage and increase physical storage consumpition
User data is never inserted into the performance tier
Reads always come from the cloud tier
Storage efficiency savings result from inline data reduction operations only
Automatic background deduplication and background scans are not available
Backup and DR processes read data from cloud tier
Other FabricPool enhancements include:
A FabricPool aggregate can store twice as much data
1:20 performance-to-cloud tier ratio limit removed
Subscription licenses are replacing perpetual license for cloud tiering
1- and 3-year license terms offered
Term-based licenses provide more customer flexibility and enable OPEX accounting
Perpetual license option will be removed from June quote tool
FlexGroup volumes also got some simplicity enhancements. Starting in ONTAP 9.6, you can rename a FlexGroup without having to re-create it and shrink a FlexGroup volume at the FlexGroup level.
Additionally, elastic sizing provides a way for file writes to complete when a member volume fills up by borrowing space from other member volumes without admin intervention.
Feature Enhancements – Data Protection and Security
In addition to simplicity and productivity, ONTAP 9.6 also brings some enhancements to data protection and security.
In-flight encryption support
Both SnapMirror (synchronous and asynchronous) and FlexCache now support TLS 1.2 encryption for communication in-flight. This can help prevent man-in-the-middle attacks and protect data contents from being viewed while being transferred.
Per-tenant/SVM key management
Previously in ONTAP, only one key manager could be used for a cluster. This was not ideal for admins who wanted to host multiple tenants on the same cluster, but provide different key management options to each. In ONTAP 9.6, each SVM can have its own unique key management system (off-box only).
Synchronous to asynchronous SnapMirror cascades (“cascade” is multiple SnapMirrors of the same volume chained together)
Synchronous replication from primary to secondary for DR
Asynchronous replication from secondary to tertiary for backup/DR
FlexCache Enhancements
FlexCache was also re-introduced in ONTAP 9.5 and offers a way for storage admins to create site-local caches of volumes to accelerate workloads at remote sites or within the same cluster.
In addition to the aforementioned in-flight encryption, the feature was expanded to add:
Qtree and quota enforcement support
100 cache volumes per node
Cloud Volumes ONTAP support
MetroCluster over IP Enhancements
MetroCluster over IP allows ONTAP clusters to stretch over a distance of 700km via standard ethernet networks. ONTAP 9.6 adds support for entry level platforms like the A220 and FAS2750, as well as support for shared inter-site links.
Welcome to the Episode 189, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the podcast, we give you the lowdown on the latest ONTAP 9.6 release with ONTAP Systems Group Vice President Octavian Tanase (@octav), Senior Director of Product Management Jeff Baxter (@baxontap), and Technical Product Marketing Manager Skip Shapiro (skip.shapiro@netapp.com)!
Join us as we talk about how ONTAP 9.6 brings more simplicity, productivity, customer use cases, data protection and security to your datacenter.
Finding the Podcast
You can find this week’s episode here:
Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.
Thoughts on Industrial Computing. Or sometimes economics or music or trail runnning or carpentry. Employee of Datto. Proud New Englander and Schaghticoke. Site is my own.
