Behind the Scenes: Episode 193 – Quarterly Security Update – Spring 2019

Welcome to the Episode 193, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

 

This week on the podcast, we deliver our quarterly security update – focusing on ONTAP 9.6 – with members of our security team.

Featured in this podcast:

  • Juan Mojica, Security PM (@juan_m_mojica)
  • Andrae Middleton, Security PM (location undisclosed)
  • Dan Tulledge, Security TME (@dan_tulledge)
  • Matt Trudewind, Security TME (@ntapmatt)

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Advertisements

Behind the Scenes: Episode 169 – Quarterly Security Update: Winter 2018

Welcome to the Episode 169, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we continue our Quarterly Security Update series on the podcast with Security PM Andrae Middleton (andrae@netapp.com) and Security TMEs Matt Trudewind (@ntapmatt) and Dan Tulledge (@Dan_Tulledge). Join us as we bring the latest information about security in NetApp products, as well as discuss security trends, breaches and other news. 

Be sure to also check out netapp.com/security, as well as the following Technical Reports: 

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

ONTAP 9.4 is now GA!

ONTAP 9 is on a new cadence model, which brings a new release every 6 months.

Today, ONTAP 9.4GA is available here!

http://mysupport.netapp.com/NOW/download/software/ontap/9.4

sully-hearye

Also, check out the documentation center:

docs.netapp.com/ontap-9/index.jsp

NetApp published a general overview blog on NVMe with Joel Reich here:

https://blog.netapp.com/the-future-is-here-ai-ready-cloud-connected-all-flash-storage-with-nvme/

Jeff Baxter’s blog is here:

https://blog.netapp.com/netapp-ontap-9-4-is-ga-modernize-your-it-architecture-with-cloud-connected-flash/

We also did an overview podcast:

And a few others on ONTAP 9.4:

And a lightboard video:

This is a brief list of the new features…

Cloud!

Fabric Pool enhancements include:

  • Tiering to Microsoft Azure Blob storage
  • Tiering of active file system data
  • Predictive performance with the object storage profiler
  • Predictive space savings for Fabric Pool before you enable it

Efficiency!

  • Increased Snapshot limits – 1,023 per volume!
  • Background aggregate level deduplication scanner
  • Automatic enable of all storage efficiencies on data protection volumes
  • Support for 30TB SAS attached SSDs
  • Deduplication across snapshots and active file system
  • Reduced node root volume sizes on new platforms

Performance!

  • End-to-end NVMe support (NVMe attached drives in the new A800 + NVMe over fibre channel support)
  • SMB multichannel support
  • 100GbE Ethernet support

Security!

  • Secure purge of files (crypto-shred!)
  • Secure boot via UEFI on new platforms
  • Validated ONTAP images
  • Protected controller reboot

I went into more detail about the features in the RC blog here:

ONTAP 9.4RC1 is now available!

For more information, check out these brief videos for some lightboard action on new ONTAP 9.4 stuff:

Some other information on the launch can be found as follows:

GCP Cloud Volumes for NFS with native access to the GCP tool suite (Google Cloud)
https://blog.netapp.com/sweet-new-storage-service-from-netapp-for-google-cloud-platform/ 

Storage Grid Update 11.1
https://blog.netapp.com/storagegrid-11-1-and-netapp-hci-the-perfect-one-two-punch-for-scaling-your-environment/ 

A800 and the A220
https://blog.netapp.com/the-future-is-here-ai-ready-cloud-connected-all-flash-storage-with-nvme/ 

ONTAP 9.4 with first to market NVMe/FC support
http://www.demartek.com/Demartek_NetApp_Broadcom_NVMe_over_Fibre_Channel_Evaluation_2018-05.html

Behind the Scenes: Episode 140 Quarterly Security Update: ONTAP 9.4 and GDPR

Welcome to the Episode 140, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we bring in Security PM Juan Mojica (@Juan_M_Mojica) and Security TMEs Andrae Middleton and Dan Tulledge to get ready for GDPR by discussing ONTAP 9.4’s newest security enhancements and what they mean for the new European regulation as the grace period ends. We also discuss best practices and how to best protect your storage systems from breaches.

For our GDPR landing page: https://www.netapp.com/us/info/gdpr.aspx

For the latest ONTAP 9.4 Security blog: https://blog.netapp.com/new-data-security-and-privacy-features-in-ontap-9-4

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 138 – ONTAP 9.4 General Overview

Welcome to the Episode 138, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, ONTAP 9.4 is here! I managed to snag ONTAP Senior Vice President Octavian Tanase (@octav) and ONTAP Chief Evangelist Jeff Baxter (@baxontap) to discuss what went into some of the decisions we made regarding the new feature payload, the future vision for ONTAP and what new stuff you can expect in this release.

To download the new release:

http://mysupport.netapp.com/NOW/download/software/ontap/9.4RC1

Check out these videos for some lightboard action on new ONTAP 9.4 stuff:

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

ONTAP 9.4RC1 is now available!

Hear ye! Hear ye! All ye storage admins! ONTAP 9.4RC1 is announced today!

sully-hearye

That’s right! Every 6 months, without fail, a new ONTAP version with a payload of new features is released.

You can find ONTAP 9.4RC1 here:

http://mysupport.netapp.com/NOW/download/software/ontap/9.4RC1

For info on what a release candidate is, see:

http://mysupport.netapp.com/NOW/products/ontap_releasemodel/

Also, check out the documentation center:

docs.netapp.com/ontap-9/index.jsp

NetApp published a general overview blog on NVMe with Joel Reich here:

https://blog.netapp.com/the-future-is-here-ai-ready-cloud-connected-all-flash-storage-with-nvme/

Stay tuned for a more general ONTAP 9.4 overview blog on the official site. Also, I recorded a brief 5-minute teaser/trailer for ONTAP 9.4 features and podcasts coming soon. Find that here:

Also a new lightboard video! Watch me write… BACKWARDS???

This blog is intended to go a little deeper into the main features available in ONTAP 9.4. We’ll break them down as follows:

  • Cloud
  • Performance
  • Efficiency
  • Security
  • General ONTAP Goodness

Without further ado…

Cloud!

FabricPools were introduced in ONTAP 9.2 as a way to tier blocks from your performance tier solution to a capacity tier, such as cloud or StorageGrid.

We covered FabricPools in detail in episode 92 of the Tech ONTAP Podcast, which you can find here:

In ONTAP 9.4, the first major updates to the feature have been released! FabricPools in ONTAP 9.4 bring the following…

Tiering cold data from the active file system

Prior to ONTAP 9.4, FabricPools only tiered cold data from snapshots on primary systems and data protection volumes on secondary systems. This allowed ONTAP to free up valuable real estate on flash systems for data actively being used. In ONTAP 9.4, inactive blocks can now be tiered off to cloud or StorageGrid from the active file system. ONTAP does this automatically by way of a new “auto” tiering policy, which has a configurable cooling period of 2-63 days (-tiering-minimum-cooling-days option in CLI). This cooling period determines how long ONTAP will wait before tiering off data considered “cool” by the policy to the FabricPool tiering destination. The tiering destination choices used to be only Amazon S3 and StorageGrid, but ONTAP 9.4 brings us…

Tiering to Azure Blob Storage

Support for Azure Blob storage was added to ONTAP 9.4 for FabricPools, which gives storage administrators more options for cloud providers. In addition, other cloud providers (such as Google Cloud, IBM Cloud Object Storage, etc) can be added via product variance requests (PVR) to your NetApp Sales reps. Keep in mind that only one cloud provider per FabricPool aggregate can be used.

fabricpools-afs

But how do you know if FabricPools will be of any value to you?

Inactive Data Reporting

Inactive Data Reporting is new in ONTAP 9.4 and can offer insight from OnCommand System Manager into whether there’s enough inactive data in your system for FabricPools to make a difference.

fabricpools-inactive-report.png

By default, this feature is enabled for aggregates participating in FabricPools, but you can also enable it via the CLI for non-FabricPool aggregates to predict space savings with the following command:

storage aggregate modify -aggregate <name> -is-inactive-data-reporting-enabled true

You can also test the performance of your FabricPool target with…

Object Store Profiler

Also new in ONTAP 9.4, the Object Store Profiler provides a way to evaluate the performance (via throughput and latency) to your desired FabricPool target. From the CLI, start the profiler using:

storage aggregate object-store profiler start -object-store-name <name> -node <name>

Then show the results with:

storage aggregate object-store profiler show

This gives a general idea of how FabricPools will work for you before you implement them.

object-profiler

But that’s not the only object store enhancements. FabricPools in ONTAP 9.4 also offers….

Better efficiency for object storage

Prior to ONTAP 9.4, there was really no concept of freeing up space on the object store once the data blocks that had been tiered off were deleted on the source. ONTAP would see the free space, but the capacity tier would not. ONTAP 9.4 offers object defragmentation for the FabricPool destination to free up deleted blocks on the destination. This is done without any admin interaction at a specific % of free space by default for different providers. The default settings are:

  • 15%Microsoft Azure Blob Storage
  • 20% Amazon S3
  • 40% StorageGRID Webscale

These percentages are adjustable via the CLI with the following command in advanced privilege:

storage aggregate object-store modify –aggregate <name> -object-store-name <name> –unreclaimed-space-threshold <%> (0%-99%)

ONTAP 9.4 also brings support for the data compaction functionality to FabricPool aggregates to provide even more storage efficiency. For more information on data compaction, see TR-4476.

What’s great about ONTAP 9.4 is that FabricPool can now be used on any ONTAP deployment (other than MCC) with…

Support for ONTAP Select and ONTAP Cloud

FabricPools can now tier from a cloud instance to a cloud tier. This is especially useful now that we have NetApp Cloud Volumes, which run on a performance tier.

Additionally, you can use FabricPools on all versions of ONTAP Select, whether standard or Premium. This means you can tier from ONTAP Select, even if it has spinning media running under the covers. This support for spinning media does not extend into FAS systems, however – just ONTAP Select. The concern there is performance; FabricPools won’t perform well on FAS systems with spinning media.

So that’s all for the FabricPool section. Now let’s talk…

Performance!

ONTAP 9.4’s biggest news is the introduction of support for NVMe over fibre channel, as well as the NVMe attached SSDs in the new AFF A800 platform. This gives NetApp the industry’s first end-to-end NVMe platform. If you’re interested in a deep dive into what NVMe is, this podcast covered it:

Early testing numbers on the new platform show sub-200 micro-second latencies, with 1.3 million IOPS per HA pair at sub-500 micro-second latencies and 34GB/s throughput. It’s a pretty beastly system.

NVMe is integral to implementaion of workloads such as machine learning and AI, which powers tech like self-driving cars, IoT devices and other budding tech.

nvme-ai.png

If you’re a NetApp employee or partner, check out the recording of the Solutions Insight Webcast from May 9 that covers NVMe in more detail.

Another performance enhancement in ONTAP 9.4 is SMB multichannel, which provides a way for SMB3 connections to leverage more TCP streams and CPU cores on the ONTAP system to increase throughput. This especially benefits SQL server workloads.

smb-multichannel.png

The new platform and ONTAP 9.4 update doesn’t just add performance, however. It also adds…

More efficiency!

The new AFF A800 platform chassis offers efficiency in the form of both power/cooling and rack space savings with >2.5PB of storage (based on a 4:72 storage efficiency ratio) in a 4U footprint. Later, when the platform supports larger NVMe attached drives, we’ll see even more density. ONTAP 9.4 also brings support for 30TB SAS attached SSDs.

But ONTAP 9.4 also brings some additional efficiencies, such as…

Snapshot block sharing

snapshot-block-share

 

Prior to ONTAP 9.4, deduplication did not take blocks locked in a snapshot under consideration for storage efficiencies. In ONTAP 9.4, if a file is locked in a snapshot *and* it exists in the active file system, deduplication will reduce the blocks needed for the file in the active file system to save even more space. ONTAP 9.4 is also adding support for up to 1,023 snapshots per FlexVol.

Background Aggregate Level Deduplication

background-aggr-dedupe

Deduplication at the aggregate level was added in ONTAP 9.2 and provides storage efficiencies when identical blocks exist across volumes in the same aggregate. This was all done inline. In ONTAP 9.4, you can now deduplicate at the aggregate level on data that’s already been placed.

Automatic Efficiency Enablement on Data Protection Volumes

auto-dedupe-schedule.png

ONTAP 9.4 also automatically enables all storage efficiencies on data protection volumes to help simplify the role of storage administrators and save space on secondary systems.

Decreased Node Root Aggregate Sizes

Every node in an ONTAP cluster has a node root aggregate, which hosts a node root volume. The node root volume holds logs, system critical files and any core files that might get generated in the event of a crash. The core file size is based on the size of system memory. As platforms add memory to systems, these core files get larger, which was causing the core files to increase, which made root volume sizes increase… wait. This is getting confusing. Here’s a diagram:

root-vol-size-equation

Advanced Disk Partitioning (or root-data partitioning) helped save some space by spreading the volume across disk partitions, but we took steps to save even more space. For example, the 1TB root aggregate that would have been needed on the A800 node gets reduced down to just 150GB!

Long story short – ONTAP 9.4 with newer systems moved the ever-increasing core files from disk media to the local flash boot storage. This applies only to newer systems (such as the A800, FAS2700 and beyond) that have large enough boot devices to hold 2 core files and cannot be retroactively applied to older systems.

ONTAP 9.4 is also bringing…

More Security!

One of the areas of ONTAP that I feel has seen some of the most significant enhancements over the past several years  has been security (credit to Juan Mojica for making it happen).

Starting with the onboard key manager, which grew into NetApp Volume Encryption and evolved into off-box key manager support and multi-factor authentication, security has grown leaps and bounds in ONTAP. This is necessary in today’s hyper-focused security minded IT organizations, as hacks, breaches and ransomware attacks are all very fresh in their minds.

ONTAP 9.4 is bringing several more security features that don’t just help guard against external threats, but also help cover internal threats (or user mistakes) from hurting a business’s bottom line.

First of all, admins can upgrade to…

Validated ONTAP Images!

validated-ontap ONTAP is now a validated image, which gives administrators peace of mind that they’re not accidentally installing some hacked version of ONTAP that can compromise their systems. In addition, it prevents engineering builds of ONTAP (which can expose clusters to undiscovered bugs or disruptions) from being used to upgrade on clusters in the field. This helps minimize the risk and exposure of running unverified builds of ONTAP.

But we’re not just protecting against upgrading to unverified installations. ONTAP 9.4 also provides…

Key-based boot technology

secure-boot

Onboard Key Manager can be leveraged to prevent reboots without a passphrase. This protects against nefarious attempts to change the admin password on a system (which can be done with console/service processor access to the boot menu of a node), as well as against physical theft of systems. In addition to the onboard key manager, you can also enable protected boot with a USB key – but you’d need a product variance request (PVR). Check with your NetApp sales rep for details. Next generation platform (yet to be released) will also provide the ability to use UEFI Secure Boot, which works in conjunction with validated ONTAP images to not only prevent upgrades to unverified ONTAP images, but from running them at all.

These provide security against external and internal threats alike, but what do you do when someone accidentally writes a classified document to a public, unclassified share

Securely purge it!

secure-purge

ONTAP 9.4 provides the ability to cryptographically shred individual files from the drive while the system remains online, and the rest of the files remain intact. This can be helpful for data spillage – e.g. when a classified document ends up in an unclassified location. This is also particularly timely and useful for the upcoming GDPR regulations’ “Right to Erasure” rules.

Security is playing a big part in the new release of ONTAP. In addition, here’s some more…

General ONTAP goodness

ONTAP 9.4 also brings several other valuable features, such as:

  • Rapid disk zeroing technology – initialize disks near-instantaneously in newer platforms!
  • 3-step, 1-click ONTAP upgrades – even easier to update your cluster non-disruptively
  • Install ONTAP without needing a separate web or FTP server
  • SQL Server support for Application Data Management in System Manager

So, there you are! A thorough rundown of the new features in ONTAP 9.4. If you feel I missed something, feel free to reach out in the comments with input!

Check out these brief videos for some lightboard action on new ONTAP 9.4 stuff:

Some other information on the launch can be found as follows:

GCP Cloud Volumes for NFS with native access to the GCP tool suite (Google Cloud)
https://blog.netapp.com/sweet-new-storage-service-from-netapp-for-google-cloud-platform/ 

Storage Grid Update 11.1
https://blog.netapp.com/storagegrid-11-1-and-netapp-hci-the-perfect-one-two-punch-for-scaling-your-environment/ 

A800 and the A220
https://blog.netapp.com/the-future-is-here-ai-ready-cloud-connected-all-flash-storage-with-nvme/ 

ONTAP 9.4 with first to market NVMe/FC support
http://www.demartek.com/Demartek_NetApp_Broadcom_NVMe_over_Fibre_Channel_Evaluation_2018-05.html

Behind the Scenes: Episode 120 – Quarterly Security Update: December 2017

Welcome to the Episode 120, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we conduct our quarterly security update with Security Product Manager Juan Mojica (@juan_m_mojica) and Information Assurance Product Manager, Mike Scanlin. We cover FIPS 140-2 validation in ONTAP, as well as what FIPS means and what the differences in levels pertain to.

For Juan’s blog, see: http://securitybrutesquad.blogspot.com 

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

ONTAP 9.3 NFS sneak preview: Mount and security tracing

aid1871175-v4-728px-trace-step-6-version-2

ONTAP 9.3 is on its way, and with it comes some long-awaited new functionality for NFS debugging, including a way to map volumes to IP addresses!

Mount trace

In ONTAP 7-Mode, you could trace mount requests with an option “nfs.mountd.trace.” That didn’t make its way into ONTAP operating in cluster mode until ONTAP 9.3. I covered a long and convoluted workaround in How to trace NFSv3 mount failures in clustered Data ONTAP.

Now, you can set different levels of debugging for mount traces via the cluster CLI without having to jump through hoops. As a bonus, you can see which data LIF has mounted to which client, to which volume!

To enable it, you would use the following diag level commands:

::*> debug sktrace tracepoint modify -node [node] -module MntTrace -level [0-20] -enabled true

::*> debug sktrace tracepoint modify -node [node] -module MntDebug -level [0-20] -enabled true

When enabled, ONTAP will log the mount trace modules to the sktrace log file, which is located at /mroot/etc/mlog/skrace.log. This file can be accessed via systemshell, or via the SPI interface. Here are a few of the logging levels:

4 – Info
5 – Error
8 – Debug

When you set the trace level to 8, you can see successful mounts, as well as failures. This gives volume info, client IP and data LIF IP. For example, this mount was done from client 10.63.150.161 to data LIF 10.193.67.218 of vserverID 10 on the /FGlocal path:

cluster::*> debug log sktrace show -node node2 -module-level MntTrace_8
Time TSC CPU:INT Module_Level
--------------------- ------------------------ ------- -------------------
 LogMountTrace: Mount access granted for Client=10.63.150.161
 VserverID=10 Lif=10.193.67.218 Path=/FGlocal

With that info, we can run the following command on the cluster to find the SVM and volume:

::*> net int show -address 10.193.67.218 -fields lif
 (network interface show)
vserver lif
------- ---------
DEMO    10g_data1

::*> volume show -junction-path /FGlocal -fields volume
vserver volume
------- ---------------
DEMO    flexgroup_local

The mount trace command can also be used to figure out why mount failures may have occurred from clients. We can also leverage performance information from OnCommand Performance Manager (top clients) and per-client stats to see what volumes might be seeing large performance increases and work our way backward to see what clients are mounting what LIFs, nodes, volumes, etc. with mount trace enabled.

Security trace (sectrace)

In ONTAP 9.2 and prior, you could trace CIFS/SMB permission issues only, using “sectrace” commands. Starting in ONTAP 9.3, you can now use sectrace on SMB and/or NFS. This is useful to troubleshoot why someone might be having access to a file or folder inside of a volume.

With the command, you can filter on:

  • Client IP
  • Path
  • Windows or UNIX name

Currently, sectrace is not supported on FlexGroup volumes, however.

cluster::*> sectrace filter create -vserver DEMO -index 1 -protocols nfs -trace-allow yes -enabled enabled -time-enabled 60

Warning: Security tracing for NFS will not be done for the following FlexGroups because Security tracing for NFS is not supported for FlexGroups: TechONTAP,flexgroupDS,flexgroup_16,flexgroup_local.
Do you want to continue? {y|n}: y

Then, I tested a permission issue.

# mkdir testsec
# chown 1301 testsec/
# chmod 700 testsec
# su user
$ cd /mnt/flexvol/testsec
bash: cd: /mnt/flexvol/testsec: Permission denied

And this was the result:

cluster::*> sectrace trace-result show -vserver DEMO

Node            Index Filter Details             Reason
--------------- ----- -------------------------- ------------------------------
node2           1     Security Style: UNIX       Access is allowed because the
                      permissions                user has UNIX root privileges
                                                 while creating the directory.
                                                 Access is granted for:
                                                 "Append"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is allowed because the
                      permissions                user has UNIX root privileges
                                                 while setting attributes.
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is allowed because the
                                                 permissions user has UNIX root privileges
                                                 while setting attributes.
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 0
                      Session-ID: -
node2           1     Security Style: UNIX       Access is not granted for:
                      permissions                "Modify", "Extend", "Delete"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /
                      Win-User: -
                      UNIX-User: 7041
                      Session-ID: -
node2           1     Security Style: UNIX       Access is not granted for:
                      permissions                "Lookup", "Modify", "Extend",
                                                 "Delete", "Read"
                      Protocol: nfs
                      Volume: flexvol
                      Share: -
                      Path: /testsec
                      Win-User: -
                      UNIX-User: 7041
                      Session-ID: -

As you can see above, the trace output gives a very clear picture about who tried to access the folder, which folder had the error and why the permission issued occurred.

Bonus Round: Block Size Histograms!

Now, this isn’t really a “new in ONTAP 9.3” thing; in fact, I found it as far back as 9.1. I just hadn’t ever noticed it before. But in ONTAP, you can see the block sizes for NFS and CIFS/SMB operations in the CLI with the following command:

cluster::> statistics-v1 protocol-request-size show -node nodename

When you run this, you’ll see the average request size, the total count and a breakdown of what block sizes are being written to the cluster node. This can help you understand your NAS workloads better.

For example, this node runs mostly a VMware datastore workload

cluster::> statistics-v1 protocol-request-size show -node node2 -stat-type nfs3_read

Node: node2
Stat Type: nfs3_read
                     Value    Delta
--------------       -------- ----------
Average Size:        30073    -
Total Request Count: 92633    -
0-511:                1950    -
512-1023:                0    -
1K-2047:              1786    -
2K-4095:              1253    -
4K-8191:             18126    -
8K-16383:              268    -
16K-32767:            4412    -
32K-65535:             343    -
64K-131071:           1560    -
128K - :             62935    -

When you run the command again, you get a delta from the last time you ran it.

If you’re interested in more ONTAP 9.3 feature information, check out Jeff Baxter’s blog here:

https://blog.netapp.com/announcing-netapp-ontap-9-3-the-next-step-in-modernizing-your-data-management/

You can also see me dress up all fancy and break down the new features at a high level here:

I’ll also be doing more detailed blogs on new features as we get closer to the release.

Behind the Scenes: Episode 109– ONTAP 9.3 Security Enhancements

Welcome to the Episode 109, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

Note: If you’re looking for last week’s podcast (IBM Watson/Elio), then it will be back up soon. It had to be reviewed before it could be officially published. Should be up as Episode 110 in a couple days.

group-4-2016

This week on the podcast, we cover the new security enhancements in ONTAP 9.3 with the security super squad, Juan Mojica (@Juan_M_Mojica, http://securitybrutesquad.blogspot.com) and Dan Tulledge (@Dan_Tulledge). Join us as we discuss Multifactor Authentication and NetApp Volume Encryption enhancements.

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Using NFSv4.x ACLs with NFSv3 in NetApp ONTAP? You betcha!

One thing I’ve come to realize from being in IT so long is that you should be constantly learning new things. If you aren’t, it’s not because you’re smart or because you know everything; it’s because you’re stagnating.

So, I was not surprised when I heard it was possible to apply NFSv4.x ACLs to files and folders and then mount them via NFSv3 and have the ACLs still work! I already knew that you could do audit ACEs from NFSv4.x for NFSv3 (covered in TR-4067), but had no idea this could extend into the permissions realm. If so, this solves a pretty big problem with NFSv3 in general, where your normal permissions are limited only to owner, group and then everyone else. That makes it hard to do any sort of granular access control for NFSv3 mounts, presents problems for some environments.

It also allows you to keep using NFSv3 for your workloads, whether for legacy application or general performance concerns. NFSv4.x has a lot of advantages over NFSv3, but if you don’t need stateful operations or the NFSv4.x features, or integrated locking, then you are safe to stay with NFSv3.

So, is it possible to use NFSv4.x ACLs with NFSv3 objects?

You betcha!

fargo-film-marge.jpg

The method for doing this is pretty straightforward.

  1. Configure and enable NFSv4.x in ONTAP and on your client
  2. Enable NFSv4.x ACL support in ONTAP
  3. Mount the export via NFSv4.x
  4. Apply the NFSv4.x ACLs
  5. Unmount and then remount the export using NFSv3 and test it out!

 

Configuring NFSv4.x

When you’re setting up NFSv4.x in an environment, there are a few things to keep in mind:

  • Client and NFS server support for NFSv4.x
  • NFS utilities installed on clients (for NFSv4.x functionality)
  • NFSv4.x configured on the client in idmapd.conf
  • NFSv4.x configured on the server in ONTAP (ACLS allowed)
  • Export policies and rules configured in ONTAP
  • Ideally, a name service server (like LDAP) to negotiate the server/client conversation of user identities

One of the reasons NFS4.x is more secure than NFSv3 is the use of user ID strings (such as user@domain.com) to help limit cases of user spoofing in NFS conversations. This ID string is required to be case-sensitive. If the string doesn’t match on both client and server, then the NFSv4.x mounts will get squashed to the defined “nobody” user in the NFSv4.x client. One of the more common issues seen with NFSv4.x mounts is the “nobody:nobody” user and group on files and folders. One of the most common causes of this is when a domain string is mismatched on the client and server.

In a client that domain string is defined in the idmapd.conf file. Sometimes, it will default to the DNS domain. In ONTAP, the v4-id-domain string should be configured to the same value on the client to provide proper NFSv4.x authentication.

Other measures, such as Kerberos encryption, can help lock the NFS conversations down further. NFSv4.x ACLs are a way to ensure that files and folders are only seen by those entities that have been granted access and is considered to be authorization, or, what you are allowed to do once you authenticate. For more complete steps on setting up NFSv4.x, see TR-4067 and TR-4073.

However, we’re only setting up NFSv4.x to allow us to configure the ACLs…

What are NFSv4.x ACLs?

NFSv4.x ACLs are a way to apply granular permissions to files and folders in NFS outside of the normal “read/write/execute” of NFSv3, and across more objects than simple “owner/group/everyone.” NFSv4.x ACLs allow administrators to set permissions for multiple users and groups on the same file or folder and treat NFS ACLs more like Windows ACLs. For more information on NFSv4.x ACLs, see:

http://wiki.linux-nfs.org/wiki/index.php/ACLs

https://linux.die.net/man/5/nfs4_acl

http://www.netapp.com/us/media/tr-4067.pdf

NFSv3 doesn’t have this capability by default. The only way to get more granular ACLs in NFSv3 natively is to use POSIX ACLs, which ONTAP doesn’t support.

Once you’ve enabled ACLs in ONTAP (v4.0-acl and/or v4.1-acl options), you can mount an NFS export via NFSv4.x and start applying NFSv4.x ACLs.

In my environment, I mounted a homedir volume and then set up an ACL on a file owned by root for a user called “prof1” using nfs4_setfacl -e (which allows you to edit a file rather than have to type in a long command).

[root@centos7 /]# mount demo:/home /mnt
[root@centos7 /]# mount | grep mnt
demo:/home on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.193.67.225,local_lock=none,addr=10.193.67.237)

The file lives in the root user’s homedir. The root homedir is set to 755, which means anyone can read them, but no one but the owner (root) can write to them.

drwxr-xr-x 2 root root 4096 Jul 13 10:42 root

That is, unless, I set NFSv4.x ACLs to allow a user full control:

[root@centos7 mnt]# nfs4_getfacl /mnt/root/file
A::prof1@ntap.local:rwaxtTnNcCy
A::OWNER@:rwaxtTnNcCy
A:g:GROUP@:rxtncy
A::EVERYONE@:rxtncy

I can also see those permissions from the ONTAP CLI:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: NFSV4 Security Descriptor
 Control:0x8014
 DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 ALLOW-OWNER@-0x1601bf
 ALLOW-GROUP@-0x1200a9-IG
 ALLOW-EVERYONE@-0x1200a9

I can also expand the mask to translate the hex:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file -expand-mask true

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: 0x20
 ...0 .... .... .... = Offline
 .... ..0. .... .... = Sparse
 .... .... 0... .... = Normal
 .... .... ..1. .... = Archive
 .... .... ...0 .... = Directory
 .... .... .... .0.. = System
 .... .... .... ..0. = Hidden
 .... .... .... ...0 = Read Only
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: NFSV4 Security Descriptor
 Control:0x8014

1... .... .... .... = Self Relative
 .0.. .... .... .... = RM Control Valid
 ..0. .... .... .... = SACL Protected
 ...0 .... .... .... = DACL Protected
 .... 0... .... .... = SACL Inherited
 .... .0.. .... .... = DACL Inherited
 .... ..0. .... .... = SACL Inherit Required
 .... ...0 .... .... = DACL Inherit Required
 .... .... ..0. .... = SACL Defaulted
 .... .... ...1 .... = SACL Present
 .... .... .... 0... = DACL Defaulted
 .... .... .... .1.. = DACL Present
 .... .... .... ..0. = Group Defaulted
 .... .... .... ...0 = Owner Defaulted

DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .1.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...1 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...1 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .1.. = Append
 .... .... .... .... .... .... .... ..1. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-OWNER@-0x1601bf
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .1.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...1 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...1 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .1.. = Append
 .... .... .... .... .... .... .... ..1. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-GROUP@-0x1200a9-IG
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .0.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...0 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...0 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .0.. = Append
 .... .... .... .... .... .... .... ..0. = Write
 .... .... .... .... .... .... .... ...1 = Read

ALLOW-EVERYONE@-0x1200a9
 0... .... .... .... .... .... .... .... = Generic Read
 .0.. .... .... .... .... .... .... .... = Generic Write
 ..0. .... .... .... .... .... .... .... = Generic Execute
 ...0 .... .... .... .... .... .... .... = Generic All
 .... ...0 .... .... .... .... .... .... = System Security
 .... .... ...1 .... .... .... .... .... = Synchronize
 .... .... .... 0... .... .... .... .... = Write Owner
 .... .... .... .0.. .... .... .... .... = Write DAC
 .... .... .... ..1. .... .... .... .... = Read Control
 .... .... .... ...0 .... .... .... .... = Delete
 .... .... .... .... .... ...0 .... .... = Write Attributes
 .... .... .... .... .... .... 1... .... = Read Attributes
 .... .... .... .... .... .... .0.. .... = Delete Child
 .... .... .... .... .... .... ..1. .... = Execute
 .... .... .... .... .... .... ...0 .... = Write EA
 .... .... .... .... .... .... .... 1... = Read EA
 .... .... .... .... .... .... .... .0.. = Append
 .... .... .... .... .... .... .... ..0. = Write
 .... .... .... .... .... .... .... ...1 = Read

In the above, I gave prof1 full control over the file. Then, I mounted via NFSv3:

[root@centos7 /]# mount -o nfsvers=3 demo:/home /mnt
[root@centos7 /]# mount | grep mnt
demo:/home on /mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.193.67.219,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.193.67.219)

When I become a user that isn’t on the NFSv4.x ACL, I can’t write to the file:

[root@centos7 /]# su student1
sh-4.2$ cd /mnt/root
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwxr-xr-x 1 root bin 0 Jul 13 10:23 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

sh-4.2$ touch file
touch: cannot touch ‘file’: Permission denied
sh-4.2$ rm file
rm: remove write-protected regular empty file ‘file’? y
rm: cannot remove ‘file’: Permission denied

When I change to the prof1 user, I have access to do whatever I want, even though the mode bit permissions in v3 say I can’t:

[root@centos7 /]# su prof1
sh-4.2$ cd /mnt/root
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwxr-xr-x 1 root bin 0 Jul 13 10:23 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

sh-4.2$ vi file
sh-4.2$ cat file
NFSv4ACLS!

When I do a chmod, however, nothing seems to change from the NFSv4 ACL for the user. I set 700 on the file, which shows up in NFSv3 mode bits:

sh-4.2$ chmod 700 file
sh-4.2$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jul 13 10:42 .
drwxrwxrwx 11 root root 4096 Jul 10 10:04 ..
-rwx------ 1 root bin 11 Aug 11 09:58 file
-rwxr-xr-x 1 root root 0 Mar 29 11:37 test.txt

But notice how the prof1 user still has full control:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 700
 UNIX Mode Bits in Text: rwx------
 ACLs: NFSV4 Security Descriptor
 Control:0x8014
 DACL - ACEs
 ALLOW-user-prof1-0x1601bf
 ALLOW-OWNER@-0x1601bf
 ALLOW-GROUP@-0x120088-IG
 ALLOW-EVERYONE@-0x120088

This is because of an ONTAP option known as “ACL Preservation.”

ontap9-tme-8040::*> nfs show -vserver DEMO -fields v4-acl-preserve
vserver v4-acl-preserve
------- ---------------
DEMO enabled

When I set the option to enabled, the NFSv4.x ACLs will survive mode bit changes. If I disable the option, the ACLs get blown away when a chmod is done:

ontap9-tme-8040::*> nfs modify -vserver DEMO -v4-acl-preserve disabled

ontap9-tme-8040::*> nfs show -vserver DEMO -fields v4-acl-preserve
vserver v4-acl-preserve
------- ---------------
DEMO disabled


[root@centos7 root]# chmod 755 file

And the ACLs are wiped out:

ontap9-tme-8040::*> vserver security file-directory show -vserver DEMO -path /home/root/file

Vserver: DEMO
 File Path: /home/root/file
 File Inode Number: 8644
 Security Style: unix
 Effective Style: unix
 DOS Attributes: 20
 DOS Attributes in Text: ---A----
Expanded Dos Attributes: -
 UNIX User Id: 0
 UNIX Group Id: 1
 UNIX Mode Bits: 755
 UNIX Mode Bits in Text: rwxr-xr-x
 ACLs: -

I’d personally recommend setting that option to “enabled” if you want to do v3 mounts with v4.x ACLs.

So, there you have it… a new way to secure your NFSv3 mounts!