XCP has traditionally been thought of as a way to rapidly migrate large amounts of data, or to scan data and generate reports. And those ideas still hold up today….
But what if i told you that you could use XCP to delete millions of files 5-6x faster than running rm on an NFS client?
Wait… why would I delete millions of files?
Normally, you wouldn’t. But in some workflows, such scratch space, this is what happens. A bunch of small files get generated and then deleted once the work is done.
I ran a simple test in my lab where I had a flexgroup volume with ~37 million files in it.
::*> vol show -vserver DEMO -volume flexgroup_16 -fields files-used vserver volume files-used ------- ------------ ---------- DEMO flexgroup_16 37356098
I took a snapshot of that data so I could restore it later for XCP to delete and then ran rm -rf on it from a client. It took 20 hours:
# time rm -rf /flexgroup/* real 1213m4.652s user 1m39.703s sys 41m16.978s
Then I restored the snapshot and deleted the same ~37 million files using XCP. That took roughly 3.5 hours:
# time xcp diag -rmrf 10.193.67.219:/flexgroup_16 real 218m17.765s user 149m16.132s sys 40m47.427s
So, if you have a workflow that requires you to delete large amounts of data that normally takes you FOREVER, try XCP next time…
These are VMs with limited RAM and 1GB network connections, so I’d imagine with bigger, beefier servers, those times could come down a bit more. But in an apples to apples test, XCP wins again!
We delete millions of files every single day – sometimes multiple times per day. Welcome to life in the EDA world.
LikeLiked by 2 people
Pingback: What you might miss about NetApp from Aug-Nov 2019, including Insight? – Why do you like WAFLs?
If using XCP to delete files in production environment, it’s highly recommended to closely monitor latency 😉
LikeLike
We delete files in production frequently. if I have to use XCP as a standard , then do I need to renew scp license every 3 months ?
LikeLike
Yes. It’s free and as easy as requesting a new license key that you copy over the old one and run “xcp activate”.
LikeLiked by 1 person
Pingback: How to Map File and Folder Locations to NetApp ONTAP FlexGroup Member Volumes with XCP | Why Is The Internet Broken?