Behind the Scenes: Episode 229 – Veeam 10

Welcome to the Episode 229, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, Michael Cade (@michaelcade1) and Adam Bergh (@ajbergh) from Veeam join myself and TME Jeannine Walter (@j9walter) to discuss the latest release of Veeam.

For more information on Veeam 10:

https://www.veeam.com/blog/v10-top-new-features-vanguards.html

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 229: Veeam 10 – Podcast Transcription

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 228 – FlexPod for Industry Verticals – Healthcare

Welcome to the Episode 228, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, we discuss FlexPod and the new initiative to create validated designs for industry verticals. First up – Healthcare and Epic software with NetApp Sr. Product Manager for Converged Infrastructure, Ketan Mota (ketan.mota@netapp.com) and NetApp Solutions Architect for Healthcare, Brian O’Mahony (omahony@netapp.com).

For links to the FlexPod technical reports:

FlexPod for Epic TRs

FlexPod for MEDITECH TRs

And general FlexPod information:

https://flexpod.com/

https://www.cisco.com/c/en/us/solutions/design-zone/data-center-design-guides/flexpod-design-guides.html

https://www.netapp.com/us/products/converged-systems/flexpod-converged-infrastructure.aspx

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 228: FlexPod for Industry Verticals: Healthcare – Transcription

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 227 – Pacific Biosciences, ONTAP and Unstructured NAS

Welcome to the Episode 227, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, Adam Knight (@damknight) of Pacific Biosciences joins us to discuss how PacBio uses ONTAP for all of its unstructured NAS workload requirements, with a focus on FlexGroup volumes!

pacbio

Also, check out these other podcast episodes:

Behind the Scenes: Episode 126 – Komprise

Behind the Scenes: Episode 209 – Designing an End-to-End Genomics Solution Using NetApp

And if you want to review the Insight presentation that Adam and I did together, check it out here (requires login):

NetApp Insight 2019 Presentations

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 227: Pacific Biosciences, ONTAP and Unstructured NAS – Transcription

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 213 – Quarterly Security Update Fall 2019

Welcome to the Episode 213, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

This week on the podcast, we continue our quarterly security series with some updates about Insight 2019, what’s new in security since the summer and some in-depth discussion about NetApp ONTAP’s Secure Purge Feature.  

Featured in this week’s podcast: 

As mentioned in the podcast, here are a few reference links you can learn from: 

https://blog.netapp.com/protecting-your-data-perfect-forward-secrecy-pfs-with-netapp-ontap/ 

https://www.youtube.com/watch?v=vL6LZmarEj4  

Achieve a Data-Centric Approach to Zero Trust with NetApp ONTAP

Network Vulnerability Scanner indicates ONTAP as a FreeBSD system

Security Hardening Guide for NetApp ONTAP 9

Secure Purge Blog

Insight Sessions: 

  • 1346-2 – ONTAP Hardening: Secure Your Data Against Today’s Threat Landscape 
  • 1347-2 – Applying a Zero Trust Architecture to Your Organization’s Security with NetApp 
  • 1348-2 – Deploying a Secure Hybrid Cloud Environment: Harness the Power of the Cloud Securely 
  • 1350-2 – Addressing Ransomware in 2019 
  • 3006-1 – Data Caretakers: Governance for Data Driven Organizations 

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 212 – Azure NetApp Files

Welcome to the Episode 212, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

This week on the podcast,  Lee Jiles (lee.jiles@netapp.comSr. Manager of Azure Global) and Will Stowe (wstowe@netapp.com, Cloud Architect) tell us all about the NetApp cloud native offering in Microsoft Azure, why you might want to use it, and how easy it is to get access. For more information, including Azure NetApp Files performance, visit https://cloud.netapp.com/azure-netapp-files. 

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Using XCP to delete files en masse: A race against rm

superman-flash-race-dc-comics-featured-image

XCP has traditionally been thought of as a way to rapidly migrate large amounts of data, or to scan data and generate reports. And those ideas still hold up today….

But what if i told you that you could use XCP to delete millions of files 5-6x faster than running rm on an NFS client?

Wait… why would I delete millions of files?

Normally, you wouldn’t. But in some workflows, such scratch space, this is what happens. A bunch of small files get generated and then deleted once the work is done.

I ran a simple test in my lab where I had a flexgroup volume with ~37 million files in it.

::*> vol show -vserver DEMO -volume flexgroup_16 -fields files-used
vserver volume files-used
------- ------------ ----------
DEMO flexgroup_16 37356098

I took a snapshot of that data so I could restore it later for XCP to delete and then ran rm -rf on it from a client. It took 20 hours:

# time rm -rf /flexgroup/*

real 1213m4.652s
user 1m39.703s
sys 41m16.978s

Then I restored the snapshot and deleted the same ~37 million files using XCP. That took roughly 3.5 hours:

# time xcp diag -rmrf 10.193.67.219:/flexgroup_16
real 218m17.765s
user 149m16.132s
sys 40m47.427s

So, if you have a workflow that requires you to delete large amounts of data that normally takes you FOREVER, try XCP next time…

These are VMs with limited RAM and 1GB network connections, so I’d imagine with bigger, beefier servers, those times could come down a bit more. But in an apples to apples test, XCP wins again!

Windows NFS? WHO DOES THAT???

Image result for disgusted girl meme

Believe it or not, Windows NFS is a thing. Microsoft has its own NFS server and client, which can leverage RFC compliant NFSv3 calls to a Windows Server running NFS server or to a 3rd party NFS server, such as NetApp ONTAP. It’s actually so popular, that NetApp had to re-introduce it in clustered ONTAP (it wasn’t there until ONTAP 8.2.3/8.3.1).

While Windows NFS currently provides NFSv3 clients, they don’t have NFSv4.1 clients – yet. They do provide NFSv4.1 as a server option, though:

https://docs.microsoft.com/en-us/windows-server/storage/nfs/nfs-overview

I cover Windows NFS support in TR-4067 starting on page 116. I am bringing this topic up because it has come up again recently and I wanted to create a quick and easy blog to follow, as well as call out how you can integrate AD LDAP to help identity management.

There are a few things you have to do to get it working in ONTAP.

Specifically:

  • enable -v3-ms-dos-client option on the NFS server
  • enable -showmount on the NFS server – this prevents some weirdness with writing files
  • disable -enable-ejukebox and -v3-connection-drop

The command would look like this:

cluster::> set advanced
cluster::*> nfs server modify -vserver DEMO -v3-ms-dos-client enabled -v3-connection-drop disabled -enable-ejukebox false -showmount enabled
cluster::*> nfs server show -vserver DEMO -fields v3-ms-dos-client,v3-connection-drop,showmount,enable-ejukebox
vserver enable-ejukebox v3-connection-drop showmount v3-ms-dos-client
------- --------------- ------------------ --------- ----------------
DEMO false disabled enabled enabled

Once that’s done, you can mount via NFS inside Windows clients using the standard “mount” command, provided you’ve enabled the Services for UNIX functionality. There’s plenty of documentation out there for that.

Just by doing the above, here’s an example of a working NFS mount in Windows:

C:\Users\Administrator>mount DEMO:/flexvol X:
X: is now successfully connected to DEMO:/flexvol

The command completed successfully.

Here’s the cluster’s view of that connection:

ontap9-tme-8040::*> network connections active show -node ontap9-tme-8040-0* -service nfs*,mount -remote-ip 10.193.67.236
              Vserver   Interface         Remote
      CID Ctx Name      Name:Local Port   Host:Port            Protocol/Service
--------- --- --------- ----------------- -------------------- ----------------
Node: ontap9-tme-8040-02
2968991376  4 DEMO      data:2049         oneway.ntap.local:931
                                                               TCP/nfs

When I write a file to the mount, there is something that can prove to be an issue, however. Users other than Administrator will write as UID/GID of 4294967294 (-2).

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /flexvol/student1-nfs.txt

                Vserver: DEMO
              File Path: /flexvol/student1-nfs.txt
      File Inode Number: 1606599
         Security Style: unix
        Effective Style: unix
         DOS Attributes: 20
DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
           UNIX User Id: 4294967294
          UNIX Group Id: 4294967294
         UNIX Mode Bits: 755
UNIX Mode Bits in Text: rwxr-xr-x
                   ACLs: -

That means users won’t show up properly/as desired in UNIX NFS mounts. For example, this is that same file from CentOS:

[root@centos7 /]# cd flexvol
[root@centos7 flexvol]# ls -la | grep student1-nfs
-rwxr-xr-x 1 4294967294 4294967294 0 Feb 5 09:18 student1-nfs.txt

So, how does one fix that?

Configuring Windows NFS clients to negotiate users properly

There are a few ways to have users leverage UID/GID other than -2.

One way is to “squash” every NFS user to the same UID/GID via the old Windows standby – the Windows registry. This is useful if only a single user will be using an NFS client.

This covers how to do that:

https://blogs.msdn.microsoft.com/saponsqlserver/2011/02/03/installation-configuration-of-windows-nfs-client-to-enable-windows-to-mount-a-unix-file-system/

Some of the third party NFS clients (such as Cygwin and Hummingbird/OpenText) will provide local passwd and group file functionality to allow you to leverage more users. In some cases, all this does is add more registry entries.

Another was is to chmod/chown the file after it’s written. But that’s not ideal.

The best way is to leverage an existing name service (such as NIS or LDAP) and have Windows clients query for the UID and GID. If you have one already, great! It’s super easy to set up the client. Just run the following command as an administrator in cmd. My NTAP.LOCAL domain already has an LDAP server set up:

C:\Users\administrator>nfsadmin mapping WIN7-CLIENT config adlookup=yes addomain=NTAP.LOCAL

The settings were successfully updated.

Once I did that, I wrote a new file and the UID/GID was properly represented:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /flexvol/prof1-nfs.txt

                Vserver: DEMO
              File Path: /flexvol/prof1-nfs.txt
      File Inode Number: 1606600
         Security Style: unix
        Effective Style: unix
         DOS Attributes: 20
DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
           UNIX User Id: 1100
          UNIX Group Id: 1101
         UNIX Mode Bits: 755
UNIX Mode Bits in Text: rwxr-xr-x
                   ACLs: -

ontap9-tme-8040::*> getxxbyyy getpwbyname -node ontap9-tme-8040-01 -vserver DEMO -username prof1
  (vserver services name-service getxxbyyy getpwbyname)
pw_name: prof1
pw_passwd:
pw_uid: 1100
pw_gid: 1101
pw_gecos:
pw_dir:
pw_shell:

If you’re interested, a packet trace shows that the Windows client will communicate via encrypted LDAP to query the user’s UNIX attribute information:

windows-ldap

An added bonus of having Windows clients query LDAP for UNIX user names and groups for NFS on ONTAP is that if you’re using NTFS security style volumes, you won’t have issues connecting to those mounts.

What breaks when doing NTFS security style?

When a UNIX user attempts to access a volume with NTFS security style ACLs, ONTAP will attempt to map that user to a valid Windows user to make sure Windows ACLs can be calculated. (I cover this in Mixed perceptions with NetApp multiprotocol NAS access)

If a user comes in with the default Windows NFS ID of 4294967294 (which doesn’t translate to a UNIX user), this is what happens.

  • The UNIX user 4294967294 tries to access the mount.
  • ONTAP receives a UID of 4294967294 and attempts to map that to a Windows user
  • That Windows user does not exist, so access is denied. This can manifest as an error (such as when writing a file) or it could just show no files/folder.

windows-nfs-ntfs-noaccess.png

windows-nfs-ntfs-noaccess2

That particular folder does have data. It’s just that the user can’t see it:

windows-nfs-ntfs-data-list

In ONTAP, we’d see this error, confirming that the user doesn’t exist:

2/5/2019 14:31:26 ontap9-tme-8040-02
ERROR secd.nfsAuth.problem: vserver (DEMO) General NFS authorization problem. Error: Get user credentials procedure failed
[ 15 ms] Hostname found in Name Service Cache
[ 19] Hostname found in Name Service Cache
[ 23] Successfully connected to ip 10.193.67.236, port 389 using TCP
**[ 28] FAILURE: User ID '4294967294' not found in UNIX authorization source LDAP.
[ 28] Entry for user-id: 4294967294 not found in the current source: LDAP. Ignoring and trying next available source
[ 29] Entry for user-id: 4294967294 not found in the current source: FILES. Entry for user-id: 4294967294 not found in any of the available sources
[ 44] Unable to get the name for UNIX user with UID 4294967294

With LDAP involved, access to the access to the NFS mounted volume with NTFS security works much better, because ONTAP and the client agree that user 1100 is prof1.

windows-nfs-ntfs-data-list-ldap

So, uh… what if I don’t have LDAP or NIS?

Well, in a Windows domain, you ALWAYS have an LDAP server. Active Directory leverages LDAP schemas to store information and any version of Windows Active Directory can be used to look up UNIX users and groups. In fact, the newer versions of Windows make this very easy. In older Windows versions, you had to manually extend the LDAP schema to provide UNIX attributes. Now, UNIX attributes like UID, UIDnumber, etc. are all in LDAP by default. All you have to do is populate these values with information. You can even do it via PowerShell CMDlets!

Once you have a working Active Directory LDAP environment, you can then configure ONTAP to communicate with LDAP for UNIX identities and you’re well on your way to having a scalable, functional multiprotocol NAS environment.

The one downside I’ve found with Windows NFS is that it doesn’t always play nicely when you want to use SMB on the same client. Windows gets a bit… confused. I haven’t dug into that a ton, but I’ve seen it enough to express caution. 🙂

Behind the Scenes: Episode 165 – Accelerate your NAS Data with FlexCache

Welcome to the Episode 165, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we talk about the new iteration of ONTAP’s NAS acceleration feature, FlexCache! Join us as we discuss with NetApp’s Technical Director Pranoop Erasani (pranoop@netapp.com), FlexCache PM Shriya Paramkusam (shriya@netapp.com) and FlexCache TME Chris Hurley (@averageguyx).

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 134 – The Active IQ Story: Building a Data Pipeline for Machine Learning

Welcome to the Episode 134, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, Active IQ Technical Director Shankar Pasupathy joins us and tells us how AutoSupport’s infrastructure and backend evolved into Active IQ’s multicloud data pipeline. Learn how NetApp is using big data analytics and machine learning on ONTAP to improve the overall customer experience

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here: