Behind the Scenes – Episode 279: NetApp and Oracle – What’s New?

Welcome to the Episode 279, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, Jeff Steiner (@TweetOfSteiner) joins us to drop some Oracle knowledge on us, as well as talk about how to best implement Oracle on NetApp storage and what’s new for Oracle on NFS.

For more information:

Podcast Transcriptions

If you want an AI transcribed copy of the episode, check it out here (just set expectations accordingly):

Episode 279: NetApp and Oracle – What’s New? (Transcript)

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes – Episode 270: NetApp ONTAP File Systems Analytics

Welcome to the Episode 270, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, we do a deep dive into the new ONTAP 9.8 feature in System Manager – File Systems Analytics – and why it’s improving the lives of storage administrators.

Podcast Transcriptions

If you want an AI transcribed copy of the episode, check it out here (just set expectations accordingly):

Episode 270: NetApp ONTAP File Systems Analytics – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes – Episode 269: Medical Imaging Workloads with NetApp Solutions

Welcome to the Episode 269, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, NetApp’s Tony Turner (@tnyturner) drops by to talk about how NetApp solutions are driving successful deployments for medical imaging workloads.

For more information:

FlexPod for Medical Imaging
https://www.netapp.com/pdf.html?item=/media/19793-tr-4865.pdf

Podcast Transcriptions

If you want an AI transcribed copy of the episode, check it out here (just set expectations accordingly):

Episode 269: Medical Imaging Workloads with NetApp Solutions – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes – Episode 263: Virtualization in ONTAP – Fall 2020 Update

Welcome to the Episode 263, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, NetApp ONTAP Virtualization PM Karl Konnerth (@konnerth) and Virtualization TME Chance Bingen (@CB8MyDataCenter) join me to discuss the latest updates to ONTAP for VMware virtualization integration, as well as what sessions they recommend for NetApp Insight 2020.

For the VMware in ONTAP Best Practices TR, see here:

TR-4597: VMware vSphere with ONTAP

Check out the virtualization sessions here:

NetApp Insight 2020 Session Catalog

Podcast Transcriptions

If you want an AI transcribed copy of the episode, check it out here (just set expectations accordingly):

Episode 263: Virtualization in ONTAP – Fall 2020 Update – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes – Episode 261: How DreamWorks Animation Creates New Worlds on NetApp ONTAP

Welcome to the Episode 261, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, I was lucky enough to snag a long-time NetApp customer and engineering partner DreamWorks Animation to talk about how the studio leverages NetApp ONTAP to build some of the most creative and effects intensive CG animated films on the planet.

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 261: How DreamWorks Animation Creates New Worlds on NetApp ONTAP – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Or, click the “view transcript” button:

gong-transcript

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

New XCP 1.6.1 Version is now Available!

flash

If you use XCP (or have used it), you might have run into an issue or two on occasion – generally related to memory issues or problems with XCP sync taking too long.

The new XCP 1.6.1 release aims to fix some of those issues.

Key enhancements

Sync Performance : Identified sync bottlenecks w.r.t change handling and fixed in 1.6.1 to Improve the sync performance. Now incremental sync takes less time than baseline.  This would help performing more frequent syncs or will need less sync/cutover window.

Memory Optimization : Memory allocation was a challenge in the previous versions and used to receive “Memory allocation” errors during the large sync jobs. XCP 1.6.1 handles the memory allocation elegantly to further assist in accomplishing the sync job.

Syslog integration : Configure Syslog option to send XCP events to the remote syslog server for monitoring the XCP jobs.

Complete Features List of NetApp XCP 1.6.1

  • Sync performance improvements (NFS/SMB)
  • Memory optimization (NFS Sync)
  • File Analytics (General Availability)
  • Event logging – XCP now supports logging events to xcp_event.log file
  • Syslog support (NFS/SMB)

The XCP TME Karthik Nagalingam also wrote up a blog on XCP:

https://blog.netapp.com/data-migration-xcp

File Analytics

In XCP 1.6, a new File Analytics Engine was added to XCP. This basically configures an Apache web server, a postgresql database and adds a web GUI to view file information using an XCP listener agent. Karthik also wrote a blog on that:

https://blog.netapp.com/xcp-data-migration-software

With this feature, you can add your NFS server/volume and XCP will run periodic scans on the datasets to provide information.

xcp_fsa_add_nfs

Once you add a server, you get a list of the available exports:

xcp_fsa_shares

Once you have the shares, you can click on one and run a scan:

xcp_fsa_home

Once that’s done, you can access the Stat View and File Distribution tabs:

The stat view includes:

  • Hot/cold data size and count
  • File capacity sorted by size
  • File distribution by size
  • Directory depth
  • Directory entries
  • Accessed time
  • Information on space occupied by users

Postgresql DB

In addition to the web GUI, you also can access the postgresql database. That way you can query the database if you choose.

When XCP configures the database, a new table called xcp_cmdb is added:

# sudo -u postgres psql
psql (9.2.24)
Type "help" for help.

postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
xcp_cmdb | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
(4 rows)

You can access that database and query which tables are added:

# sudo -u postgres psql xcp_cmdb
psql (9.2.24)
Type "help" for help.

xcp_cmdb=# \dt
List of relations
Schema | Name | Type | Owner
--------+---------------------+-------+---------
public | xcp_agent | table | central
public | xcp_login | table | central
public | xcp_nfs_file_server | table | central
public | xcp_nfs_jobs | table | central
public | xcp_nfs_stats | table | central
public | xcp_nfs_tree | table | central
public | xcp_smb_file_server | table | central
public | xcp_smb_jobs | table | central
public | xcp_smb_stats | table | central
public | xcp_smb_tree | table | central
(10 rows)

You can use standard Postgresql commands to query the tables.

xcp_cmdb=# SELECT file_server_ip FROM xcp_nfs_file_server;
file_server_ip
----------------
10.10.10.10
(1 row)


xcp_cmdb=# SELECT * FROM xcp_nfs_jobs;
id | export_path | agent_id | process_group_id | start_timestamp |
end_timestamp | type | status | statistics |
stdout | stderr | log | protocol | files_to_process
--------------------------------------+-------------------------+--------------------------------------+------------------+----------------------------+-----
-----------------------+------+----------+------------------------------------------------------------------------------------------------------------------+
--------+--------+-----+----------+------------------
745a0615-4177-4d83-b945-8e302efc89e9 | 10.10.10.10:techontap | bf83aced-81cd-4b7f-be5d-94e55df11484 | 6562 | 2020-07-27 06:26:55.980298 | 2020
-07-27 06:26:56.785463 | SCAN | FINISHED | {"received": 1699564, "scanned": 6928, "sent_throughput": 138247, "sent": 106400, "receive_throughput": 2208273} |
stdout | stderr | log | NFS | 9047
9a5fc78e-7260-47dd-8cb5-da1820dbcd3e | 10.10.10.10:/home | bf83aced-81cd-4b7f-be5d-94e55df11484 | 2039 | 2020-07-31 00:34:50.001866 | 2020
-07-31 00:34:50.390331 | SCAN | FINISHED | {"received": 9808, "scanned": 21, "sent_throughput": 5627, "sent": 2072, "receive_throughput": 26638} |
stdout | stderr | log | NFS | 839
2767d48c-be49-4232-9a49-9632878bd4c6 | 10.10.10.10:techontap | bf83aced-81cd-4b7f-be5d-94e55df11484 | 5832 | 2020-07-27 06:17:47.496321 | 2020
-07-27 06:17:48.290484 | SCAN | FINISHED | {"received": 1699564, "scanned": 6928, "sent_throughput": 83533, "sent": 106400, "receive_throughput": 1334304} |
stdout | stderr | log | NFS | 9114
(3 rows)

Installation/Upgrade Note

If you’re already running an XCP version prior to 1.6.1, you’ll need to re-run the configure.sh script. This creates the XCP service module on the client (NFS version) to provide better overall management.

For example, if I simply untar the XCP install (like you would do in prior versions to upgrade), then XCP commands work fine, but the new File Systems Analytics module doesn’t, as it needs the XCP service to run properly.

# tar -xvf NETAPP_XCP_1.6.1.tgz
./
./xcp/
./xcp/linux/
./xcp/linux/xcp
./xcp/windows/
./xcp/windows/xcp.exe
./xcp/xcp_gui/
./xcp/xcp_gui/xcp/
./xcp/configure.sh
./xcp/README

# xcp activate
XCP 1.6.1; (c) 2020 NetApp, Inc.; Licensed to Justin Parisi [NetApp Inc] until Thu Oct 22 12:31:11 2020

XCP already activated

Trying to start the listener will fail:

# ./linux/xcp --listen &
[1] 1525
[root@XCP xcp]# Traceback (most recent call last):
File "../xcp.py", line 957, in <module>
File "../xcp.py", line 902, in main
File "../rest/config.py", line 117, in <module>
File "../rest/config.py", line 112, in singleton
File "../rest/config.py", line 163, in __init__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3215, in first
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3007, in __getitem__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3339, in _execute_and_instances
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3354, in _get_bind_args
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3332, in _connection_from_session
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1123, in connection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1129, in _connection_for_bind
File "/local/build/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 430, in _connection_for_bind
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2266, in _wrap_pool_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1536, in _handle_dbapi_exception_noconnection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 354, in connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
File "/local/build/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
File "/local/build/lib/python2.7/site-packages/sqlalchemy/pool/base.py", line 630, in __connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
File "/local/build/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 453, in connect
File "/local/build/lib/python2.7/site-packages/pgdb.py", line 1623, in connect
sqlalchemy.exc.InternalError: (pg.InternalError) could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?

(Background on this error at: http://sqlalche.me/e/2j85)

[1]+ Exit 1 ./linux/xcp --listen

When we look for the XCP service, it’s not there. That gets created with the script:

# service xcp status
Redirecting to /bin/systemctl status xcp.service
Unit xcp.service could not be found.
# systemctl status xcp
Unit xcp.service could not be found.
# sudo systemctl start xcp
Failed to start xcp.service: Unit not found.

Here’s an example of the configure.sh script to upgrade:

# ./configure.sh 
------------XCP CONFIGURATION SCRIPT -------------
# Checking if xcp service is running

Menu Help:
# Install/ Upgrade -
# Install - Installs PostgreSQL and httpd if not installed and does setup for XCP File Analytics.
# Upgrade - Upgrades takes a backup of xcp folder under /var/www/html/ to xcp_backup
# Repair - Restarts PostgreSQL database, http and XCP services incase if any of them are stopped.
# Cleanup - Uninstalls PostgreSQL and httpd.

1. Installation/Upgrade
2. Repair
3. Cleanup
0. Exit

Please select the option from the menu [0-3]: 1 
--------Setting up Postgres -----------
# Checking if postgres is installed
Redirecting to /bin/systemctl status postgresql.service
# Postgres service is already running 
--------------Setting up httpd -------------
# Checking if httpd is installed
httpd-2.4.6-93.el7.centos.x86_64
# HTTPD is already installed... 
--------------Setting up https -------------
mod_ssl-2.4.6-93.el7.centos.x86_64
# Https is already installed... 
--------------Configuring xcp --------------
# Copying xcp files.....
# Taking backup of xcp folder /var/www/html/xcp in /var/www/html/xcp_backup
NOTE:
If you are running for first time, select option 1.
If you are upgrading select option 0.
______________________________
Please choose the menu you want to start:
1. Configure client system
2. Reset admin password
3. Reset Postgres database password
4. Rebuild xcp.ini file
5. Delete database and reconfigure client system

0. Quit
-------------------------------
Enter digit between [0-5] >> 1

# Postgres service is running
# Database already exists and configured. 
---------------Starting xcp service -------------
# Creating xcp service...
Redirecting to /bin/systemctl status xcp.service
# XCP service started successfully
Created symlink from /etc/systemd/system/multi-user.target.wants/xcp.service to /etc/systemd/system/xcp.service.
root 1818 1 19 20:03 ? 00:00:00 /usr/bin/xcp --listen
postgres 1833 1587 0 20:03 ? 00:00:00 postgres: central xcp_cmdb 127.0.0.1(34018) idle 
---------------------------------------------
XCP File analytics is configured successfully
To start XCP service manually, use "sudo systemctl start xcp" command

Use following link and login as 'admin' user for GUI access
https://XCP/xcp 
---------------------------------------------

Once that’s done, XCP is ready to use!

Full NetApp Support

Since XCP 1.5, you can call in for official NetApp Support for XCP.

Support Site Self-Service Options

To review, order, and renew software subscriptions, click My Service Contracts & Warranties on the Service and Support page of the NetApp Support site.

New/Updated NAS Technical Reports! – Spring 2020

With the COVID-19 quarantine, stay at home orders and new 1-year ONTAP release cadence, I’m finding I have a lot more spare time, which translates into time to update old, crusty technical reports!

30 Gandalf Facts To Rule Them All | The Fact Site

Some of the old TRs hadn’t been updated for 3 years or so. Much of the information in those still applied, but overall, the TR either had to be retired or needed an update – if only to refresh the publish date and apply new templates.

So, first, let’s cover the grandfather TRs.

Updated TRs

TR-4073: Secure Unified Authentication

This TR was a monolith that I wrote when I first started as a TME back in 2015-ish. It covers LDAP, Kerberos and NFSv4.x for a unified security approach to NFS. The goal was to combine everything into a centralized document, but what ended up happening was I now had a TR that was 250+ pages long. Not only is that hard to read, but it’s also daunting enough to cause people not to want to read it at all. As a result, I made it a goal to break the TR up into more manageable chunks. Eventually, this TR will be deprecated in favor of newer TRs that are shorter and more specific.

TR-4616: NFS Kerberos in ONTAP

I created the NFS Kerberos TR in 2017 to focus only on Kerberos with NFS. To streamline the document, I narrowed the focus to only a set of configuration options (AD KDCs, RHEL clients, newest ONTAP version), removed extraneous details and moved examples/configuration steps to the end of the document. The end result – a 42 page document with the most important information taking up around 30 pages.

However, there hasn’t been an updated version since then. I’m currently in the process of updating that TR and was waiting on some other TRs to be completed before I finished this one. The new revision will include updated information and the page count will rise to around 60-70 pages.

TR-4067: NFS Best Practice Guide

This TR is another of the original documents I created and hasn’t been updated since 2017. It’s currently getting a major overhaul right now, including re-organizing the order to include the more crucial information at the start of the document and reducing the total page count by roughly 20 pages. Examples and advanced topics were moved to the back of the document and the “meat” of the TR is going to be around 90 pages.

Major changes include:

  • New TR template
  • Performance testing for NFSv3 vs. NFSv4.x
  • New best practice recommendations
  • Security best practices
  • Multiprotocol NAS information
  • Removal of Infinite Volume section
  • NFS credential information

As part of the TR-4073 de-consolidation project, TR-4067 will cover the NFSv4.x aspects.

This TR is nearly done and is undergoing some peer review, so stay tuned!

TR-4523: DNS Load Balancing in ONTAP

This TR was created to cover the DNS load balancing approaches for NAS workloads with ONTAP. It’s pretty short – 35 pages or so – and covers on-box and off-box DNS load balancing.

It was updated in May 2020 and was basically a minor refresh.

New TR

TR-4835: How to Configure LDAP in ONTAP

The final part of the TR-4073 de-consolidation effort was creating an independent LDAP TR. Unlike the NFS Kerberos TR, I wanted this one to cover a wide array of configurations and use cases, so the total length ended up being 135 pages, but the “meat” of the document (the most pertinent information) only takes up around 87 pages.

Sections include, in order:

  • LDAP overview
  • Authentication in ONTAP
  • LDAP Components and Considerations
  • Configuration
  • Common Issues and Troubleshooting
  • Best Practices
  • Appendix/Command Examples

Feedback and comments are welcome!

Mounting ONTAP CIFS/SMB shares with Linux – Guidelines and tips

Every now and then, the question of mounting CIFS/SMB shares in ONTAP using Linux clients comes up. Many people might consider this configuration to be a crime against nature.

garbodor

If you’re asking yourself “why would someone want to do this,” there are a few reasons.

  • Lock handling – SMB locks use oplocks. NFS locks use shared locks or delegations, depending on the NFS version and options being used. Generally this is fine, but in some cases, the style of lock might create complications with some applications. For example, Kafka on NFS has some issues with how NFS handles deletions when a file is locked:
    https://blog.usejournal.com/kafka-on-kubernetes-a-good-fit-95251da55837?gi=91867e828261
  • Legacy copiers/scanners – Many companies will use copiers/scanners to create image files that redirect to a NAS share in their environment. Many of those copiers use old SMB clients and are no longer under support, so they can’t be upgrades. And many of those scanners/copiers are expensive, so people aren’t interested in buying newer ones just to get newer SMB versions.
  • “How we’ve always done it” – This is a common reason that doesn’t always have technical logic associated with it, but you still have to roll with it.

There are other reasons out there as well, I’m sure. But this post will attempt to group the considerations, support statements and issues you might see in case someone else has that question.

Mounting SMB shares in ONTAP – General Guidelines

In 7-Mode ONTAP, mounting SMB shares with clients like AIX had no issues. This was because 7-Mode supported a lot of old SMB technology concepts that newer versions of ONTAP have deprecated. Even though SMB shares on Linux worked in 7-Mode, there was never an official supported configuration for it.

NetApp ONTAP has traditionally only supported Apple OS and Windows OS for CIFS/SMB workloads.

You can find the supported clients in the Interoperability Matrix here:

https://mysupport.netapp.com/matrix/

With newer versions of ONTAP, you *can* get SMB/CIFS working properly, provided you take the following into account:

  • CIFS/SMB clients should support UNICODE for CIFS/SMB communication. Clustered ONTAP doesn’t support non-UNICODE in any release and never will. Newer CIFS/SMB clients should support UNICODE. AIX is a good example of an older Linux client that has some challenges using SMB because it still uses non-UNICODE in some client installs.
  • CIFS/SMB versions mounting should reflect the supported SMB versions in ONTAP. ONTAP has deprecated SMB1, so you’d need to mount using SMB2 or SMB3.
  • CIFS/SMB enabled features in ONTAP should reflect the feature support for the CIFS/SMB Linux clients. For example, my Centos7 Linux Samba client doesn’t support large MTU, SMB3 encryption, multichannel, etc. That can cause failures in the mounts (for an example, see below).
  • If using Kerberos for the CIFS/SMB mounts, be sure the CIFS/SMB ONTAP server has a valid SPN for the CIFS service. The name should match the mount name. For example, if your CIFS/SMB server is named “CIFS” then the SPN would be cifs/cifs.domain.com.

Common Issues when Mounting CIFS/SMB

One of the main issues people run into with SMB mounts in Linux (provided they follow the guidelines above when using ONTAP) is using the wrong command syntax. For example, when you specify the share path, you use either \\\\name\\share (extra slashes to tell Linux that the \ isn’t an operand) or you use //name/share.

The following NetApp KB article does a good job of giving command examples for the right way to mount SMB:

How to access CIFS from Linux machine using SAMBA

Usually the command will tell you if the wrong option is being used, but sometimes the errors are less than helpful. These are a few errors I ran into.

Incorrect Mount Option

This was pretty self-explanatory. First, I didn’t have the cifs-utils installed. Second, I used the wrong command syntax.

Mount point does not exist

Again, self explanatory. The directory you’re trying to mount to needs to exist.

No such file or directory (when mounting)

This means you specified the wrong export path on the NFS server. Check your junction-paths and try again.

Host is down

This one was a tricky error, as it suggests that the server was not up. In some cases, that may truly be the issue. But it wasn’t in my case.

# mount -t cifs -o user=cifsuser \\\\DEMO\\nas /mnt/nas
Password for cifsuser@\DEMO\nas: **********
mount error(112): Host is down

But, in reality, the issue was that I didn’t specify the SMB version and it tried to use SMBv1.0 by default.

Specifying the SMB version (-o vers=3.0) got past that issue.

Required key not available

This is a Kerberos specific error. In my case, the cifs/servername.domain.com SPN didn’t exist for the hostname I used in the UNC path. You can see that in a packet capture.

Permission denied

This error is fairly useless in both Windows and Linux in a lot of cases – mainly because it can mean a variety of things. Sometimes, it really is an access issue (such as share or file-level permissions). But in my testing, I also ran into this issue when I had unsupported SMB features enabled on my CIFS server in ONTAP.

# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas
Password for administrator@//DEMO/nas: **********
mount error(13): Permission denied

In a packet trace, I could see the client telling me what it supported:

smb-capabilities

But the reply didn’t really mention the issue. So I made sure to disable the following CIFS/SMB features:

  • SMB3 encryption (cifs security modify)
  • Large MTU and SMB Multichannel (cifs options modify)

Once I did that, I was able to mount.

[root@centos7 /]# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas
Password for administrator@//DEMO/nas: **********
[root@centos7 /]# touch /mnt/nas/smbfile
[root@centos7 /]# ls -la /mnt/nas
total 1
drwxr-xr-x 2 root root 0 Mar 27 14:36 .
drwxr-xr-x. 9 root root 97 Mar 27 10:17 ..
-rwxr-xr-x 1 root root 0 Mar 27 14:36 smbfile

For Kerberos, you would just specify sec=krb5.

# mount -t cifs -o sec=krb5,vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas

Stale file handle errors when crossing junction paths

In some cases, if you mount volumes to volumes (via junction-path), newer Linux clients may report different inode numbers for the junctioned volume when using a command like “ls -i” than when using “ls -id”. This can create potential access problems from clients, such as “stale file handle” errors.

For example, this Centos 8.3 client shows different inode numbers for the same volume:

# ls -i
8107029540347314273 mountedvolume
# ls -id /mnt/cifs/mountedvolume/
2744251190362505280 /mnt/cifs/mountedvolume/

The way around this issue is to mount with the mount option “actimeo=0.” Newer SMB clients seem to cache inode information and create potential conflicts. Disabling actimeo prevents that caching from taking place. Since this is a caching option, you might see slower overall performance, but it will fix the stale file handle issue. Alternately, you may want to mount with the “noserverino” option instead, which doesn’t try to use the SMB server’s inodes.

Inode Numbers

When Unix Extensions are enabled, we use the actual inode number provided by the server in response to the POSIX calls as an inode number.

When Unix Extensions are disabled and "serverino" mount option is enabled there is no way to get the server inode number. The client typically maps the server-assigned "UniqueID" onto an inode number.

Note that the UniqueID is a different value from the server inode number. The UniqueID value is unique over the scope of the entire server and is often greater than 2 power 32. This value often makes programs that are not compiled with LFS (Large File Support), to trigger a glibc EOVERFLOW error as this won't fit in the target structure field. It is strongly recommended to compile your programs with LFS support (i.e. with -D_FILE_OFFSET_BITS=64) to prevent this problem. You can also use "noserverino" mount option to generate inode numbers smaller than 2 power 32 on the client. But you may not be able to detect hardlinks properly.

Once I set actimeo=0, this is how those inodes looked:

# ls -i
8107029540347314273 mountedvolume
# ls -id /mnt/cifs/mountedvolume/
8107029540347314273 /mnt/cifs/mountedvolume/

There you go! Hopefully this helped someone from going down a Google K-hole. Leave comments or suggestions below.

Getting ONTAP NFS Kerberos to Work with FreeIPA

Image result for hamburglar

Obviously, with the social distancing/lockdowns happening, I have had more time to write up blogs on things. So, here’s another one. If you have suggestions for topics, let me know and I’ll see about writing them up.

Before we get to the meat of what the title is (or scroll down to the bottom if you want the quick and dirty steps), let’s recap NFS security and Kerberos…

NFS is a protocol that allows multiple clients communicate to a single storage endpoint as a way to share data across a network. Because data is transmitted over a network, it’s important to be able to secure that data, both at-rest and in-flight.

At-rest Security in NFS

At-rest security refers to security applied to data residing on the storage system, as well as the interaction between NFS client and NFS server to negotiate things like NFS version, identity, etc.

At-rest security for NFS includes:

  • Export policies and rules
  • Permissions/ACLs
  • User/group owners
  • Name ID strings (NFSv4.0 and later)

However, when data passes over the wire, packet sniffers are able to see identifying information like IP addresses, users, groups, permissions, file names, paths, etc. All it would take is someone being able to see this information to easily create duplicate set ups to “spoof” users and groups and gain access to data. At-rest security mostly protects you from threats that don’t have this knowledge or expertise. To prevent bad actors from getting information from the transmission of data, you’d need to set up in-flight security.

In-flight Security in NFS

In-flight security refers to securing the actual data packets in transit.

For NFS, this can be done in a few ways:

For ONTAP, tunneling NFS over SSH or stunnel isn’t supported, but you can use NFS Kerberos.

For a deeper look at NFS Kerberos in ONTAP, see:

NFS Kerberos in ONTAP Primer

Kerberos Security Flavors

NFS Kerberos has 3 methods that can be used to secure things. Each provides an additional level of security, but also adds extra performance overhead.

Keep in mind that NFSv3 *can* use Kerberos, but only the NFS portion of the protocol will use it. Ancillary protocols like mountd, portmapper, NLM, etc. will still be unencrypted. The most secure version of NFS available is NFS v4.x, which combines all the NFS ancillary protocols into a single port and compound NFS calls, which can all be Kerberized.

krb5 – Encrypts the authentication only

Remember when I said if a bad actor stole information about the client, user, etc. that they could spoof that user to get access? Well, with krb5, that becomes harder to do. You can have all the information about the NFS export, but to gain access to a Kerberized mount, you’d need to have a valid username and password to get a Kerberos TGT, which would then be used to get the NFS service ticket. In addition, your client would also need to have a Kerberos ticket and keytab to gain access, so unless your attacker has KDC admin rights, they won’t be able to access the mount.

krb5i – Authentication with data checksums/integrity checking

Krb5 secures the initial authentication for NFS, but the actual NFS data packets are still transmitted in clear text across the wire. If a bad actor were to plug in and sniff those data packets in flight, they could see everything. In addition, bad actors could also act as a “man in the middle” to intercept packets and then transmit their own data if they chose. Krb5i prevents this by using checksums on the client and server to verify that all the data arriving and leaving is coming from a trusted source. The data is still visible, but it can’t be interfered with.

krb5p – Authentication/integrity checking/end to end encryption (privacy)

For the ultimate in-flight security hammer for NFS, you would use krb5p. This combines what krb5i does with in-flight encryption of all NFS packets. That means all NFS packets will be encrypted with the enctype specified in the Kerberos configuration. This can add considerable overhead for NFS performance, but will also prevent NFS packets from being sniffed easily.

How to set up NFS Kerberos in ONTAP

Setting up Kerberos in ONTAP requires a few things:

  • A Key Distribution Center (Microsoft AD, MIT Kerberos, FreeIPA, RedHat IDM)
  • DNS server or static host entries
  • Optional (but recommended): LDAP server for UNIX identity management
  • Configuring NFS clients and ONTAP Service Principal Names

The KDC is the central hub for Kerberos operations and is responsible for handing out Kerberos tickets to clients, users and services for authentication in a Kerberos realm.

DNS is used to resolve IP addresses to host names, which are then passed to the KDC as SPN requests. For example, if client 10.10.10.10 tries to get a Kerberos ticket, a DNS reverse lookup will be done to find the hostname FQDN “client1.domain.com” Then a SPN request for host/client1.domain.com@DOMAIN.COM is made to the KDC. If the SPN exists and matches the request, then the Kerberos request moves on to the next steps. If it doesn’t exist, the request fails.

LDAP is used to centralize the UNIX identities for users and groups to ensure clients and servers have the same information all the time, without manual intervention. This is less important to Kerberos and more important to NFSv4.x and NFS permission resolution.

NFS client configuration for Kerberos on newer clients is fairly straightforward; you configure DNS (so it can find the Kerberos realm name and client hostname) and then simply use one of the automated tools to “join” the Kerberos realm. This automatically creates the service principal and transfers the keytab files. Where it gets tricky is if you have to do a manual Kerberos configuration. TR-4073 and TR-4616 can be of some guidance there, as well as a bevy of articles across the web.

NFS server/ONTAP configuration for Kerberos is relatively simple; you configure DNS and the Kerberos realm and then you enable Kerberos on a network interface. When you do that, ONTAP interacts with the KDC you specified in the Kerberos realm and automates the principal creation – provided the KDC is using standard Kerberos interaction, such as Microsoft Active Directory or kadmin for Linux.

Now we’re getting to the part the title hinted about… FreeIPA Kerberos setup.

Why do I need a blog on FreeIPA Kerberos with ONTAP? You just said it was easy!

I did say it was easy – if the KDC is using standard kadmin. However, FreeIPA happens to use a wrapper over kadmin for KDC management and discourages the use of kadmin for management of service principals. In fact, by default, they lock things down pretty tight.

In FreeIPA, there is a GUI to make things simpler to use. There are also “ipa” commands to interact via the CLI, such as ipa user-add or ipa service-add. In most Linux KDCs, there was kadmin to manage things. In fact, there are *two* kadmins. One is for local management (kadmin.local) and the other is for remote management (kadmin). For more info on the differences, see:

https://docs.oracle.com/cd/E19683-01/816-0211/6m6nc66tj/index.html

The remote kadmin is what ONTAP interacts with for Linux KDCs. When you use the automated ONTAP method to add the NFS service principal, the following happens:

  • A prompt for a username and password to issue a kinit to the KDC to gain access to kadmin
  • get_principal is run via remote kadmin to see if the SPN exists
    • If the SPN doesn’t exist, create_principal is run via remote kadmin
    • If the SPN does exist, modify_principal is run via remote kadmin

All of these kadmin commands require the proper privileges to run successfully. In FreeIPA, remote kadmin is locked down by default to even run get_principal.

For example:

# ipa user-add krbadmin

---------------------
Added user "krbadmin"
---------------------
User login: krbadmin
First name: Kerberos
Last name: Admin
Full name: Kerberos Admin
Display name: Kerberos Admin
Initials: KA
Home directory: /home/krbadmin
GECOS: Kerberos Admin
Login shell: /bin/sh
Principal name: krbadmin@CENTOS-LDAP.LOCAL
Principal alias: krbadmin@CENTOS-LDAP.LOCAL
Email address: krbadmin@centos-ldap.local
UID: 1971600007
GID: 1971600007
Password: False
Member of groups: ipausers
Kerberos keys available: False

# kadmin -p krbadmin
Authenticating as principal krbadmin with password.
Password for krbadmin@CENTOS-LDAP.LOCAL:
kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Operation requires ``get'' privilege while retrieving "nfs/server.domain.com@DOMAIN.COM".

The ONTAP error we’d see is less clear, however. The error suggests that the SPN *already exists*.

Error: NFS Kerberos bind SPN procedure failed
  [  0 ms] Creating account in Unix KDC
  [    30] Successfully connected to ip 10.x.x.x, port 749
           using TCP
**[    40] FAILURE: Unexpected state: Error 1142 at
**         file:src/utils/secd_kadmin_utils.cpp
**         func:createVifKrbAccountUsingKadmin line:227
**[    40] FAILURE: spn already exists. Failed to reuse spn
**         'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
**         admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
**         Unknown code 0
  [    41] Uncaptured failure while creating account

Obviously, the error should refer to some permissions issue. When we see this error in ONTAP,  we can verify on the KDC what is happening via the /var/log/kadmind.log file. In this case, we see “unauthorized request.”

Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Request: kadm5_init, kadmin/admin@CENTOS-LDAP.LOCAL, success, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x, vers=2, flavor=6
Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Unauthorized request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

To give permissions to users to use remote kadmin, you have to modify the /var/kerberos/krb5kdc/kadm5.acl file.

In my setup, this is how I configured the ACL:

# cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@CENTOS-LDAP.LOCAL *
ontap@CENTOS-LDAP.LOCAL *
krbadmin@CENTOS-LDAP.LOCAL *

Then you restart IPA services:

# service ipa restart
Redirecting to /bin/systemctl restart ipa.service

Now, my user can query the principal:

kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Principal does not exist while retrieving "nfs/server.domain.com@DOMAIN.COM".

However, ONTAP gets a new (less descriptive) error when trying to interact with the KDC:

Error: NFS Kerberos bind SPN procedure failed
[ 1 ms] Creating account in Unix KDC
[ 29] Successfully connected to ip x.x.x.x, port 749
using TCP
**[ 45] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:createVifKrbAccountUsingKadmin line:223
**[ 45] FAILURE: Failed to create spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
** Invalid argument
[ 45] Uncaptured failure while creating account

The /var/log/kadmind.log is also fairly useless here:

Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Principal does not exist, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x
Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_create_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid argument, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

ONTAP is just reporting what kadmin says!

Our clue comes from when we try to manually create the SPN using kadmin:

kadmin: add_principal -randkey nfs/server.domain.com@DOMAIN.COM
WARNING: no policy specified for nfs/server.domain.com@DOMAIN.COM; defaulting to no policy
add_principal: Kerberos database constraints violated while creating "nfs/server.domain.com@DOMAIN.COM".

Same goes for modify_principal:

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
(vserver nfs kerberos interface enable)

Username: ontap

Password:

Warning: An account that matches the given service principal name already exists. Re-using this account deletes and re-creates the account if it is not shared by the LIFs in Vserver "NFS". This
invalidates any existing Kerberos service tickets and keys for this account. Do you want to re-use this account? {y|n}: y

Error: NFS Kerberos bind SPN procedure failed
[ 0 ms] Creating account in Unix KDC
[ 39] Successfully connected to ip x.x.x.x, port 749
using TCP
[ 45] Re-using the account for
spn=nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
**[ 53] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:randomizePasswordUsingKadmin line:287
**[ 53] FAILURE: Failed to randomize password for spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'ontap@CENTOS-LDAP.LOCAL'
[ 54] Uncaptured failure while creating account


Mar 24 11:14:54 centos8-ipa.centos-ldap.local kadmind[17208](Notice): Request: kadm5_randkey_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid key/salt tuples, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

And delete_principal:

::*> kerberos interface disable -vserver NFS -lif ipa-krb
(vserver nfs kerberos interface disable)

Username: ontap

Password:

Warning: This command deletes the service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" from the machine account on the KDC. Do you want to continue? {y|n}: y

Error: command failed: Failed to disable NFS Kerberos on LIF "ipa-krb". Failed to delete the account associated with the Kerberos service principal name. Reason: cifs smb kadmin error.

So that means, even though we gave full rights to that user in the kadm5.acl file, we can’t create principals in FreeIPA using kadmin. They want to funnel you to use the IPA tools – likely for a good reason. There’s probably a bunch of other automated steps that take place with those tools, so this is in the interest of simplicity. and security.

How do we get past this?

There are two options here.

  1. Open up the permissions on FreeIPA to use kadmin to create/add/modify/delete principals. (I haven’t figured out how to do this yet)
  2. Manually create the SPN and keytab file and then use ONTAP to transfer the keytab via URI

I went with option 2.

The TL;DR steps, as provided by Alexander Bokovoy in the comments:

- kinit admin
- ipa host-add demo-ipa.centos-ldap.local
- ipa service-add nfs/demo-ipa.centos-ldap.local
- ipa-getkeytab -p nfs/demo-ipa.centos-ldap.local -k ./nfs.keytab -e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

Remember that ONTAP only supports the following enctypes for NFS Kerberos:

  • AES-128 and AES-256
  • DES and 3DES

Then, you copy that file to your HTTP or FTP server. The address to the file will be used in the ONTAP CLI command.

Note: If you’re using IIS, you may have to allow .keytab as a MIME type.

For example:

iis-mime-keytab

Once the file is web-accessible, you run the kerberos interface enable command and use the -keytab-uri option to upload the keytab. Unsupported enctypes will get discarded.

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL -keytab-uri http://web-server/files/ipakrb-ontap.keytab
(vserver nfs kerberos interface enable)

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

Warning: Keys for encryption types "des-cbc-crc,des3-cbc-sha1,aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96" are required for Vserver "NFS" but found keys only for encryption types
"aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96". Keys for encryption types "des-cbc-crc,des3-cbc-sha1" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" are
missing. Available keys will be imported. Do you want to continue? {y|n}: y

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

You’ll also want to ensure you have a valid UNIX user or name mapping rule for the krb-spn name mapping for the NFS clients. That’s covered in TR-4616 in detail.

Once that’s all done, NFS Kerberos mounts should work fine!

[root@centos8-1 sysconfig]# mount -o nfsvers=4,minorversion=1,sec=krb5 x.x.x.x:/kerberos /mnt
[root@centos8-1 sysconfig]# mount | grep krb
x.x.x.x:/kerberos on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.y,local_lock=none,addr=x.x.x.x)
# su ipa-user
sh-4.4$ cd /mnt
sh: cd: /mnt: Permission denied
sh-4.4$ kinit
Password for ipa-user@CENTOS-LDAP.LOCAL:
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ cd /mnt
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
03/23/2020 14:40:16 03/24/2020 14:39:27 nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ touch krb5file
sh-4.4$ ls -la
total 4
drwxrwxrwx. 2 root root 4096 Mar 23 14:40 .
dr-xr-xr-x. 17 root root 224 Jan 21 10:03 ..
-rw-------. 1 ipa-user admins 0 Mar 23 14:40 krb5file
-rw-r--r--. 1 root root 0 Mar 23 14:15 v3nokrb
-rw-r--r--. 1 root root 0 Mar 23 14:16 v4nokrb
-rw-------. 1 ipa-user admins 0 Mar 23 14:22 v4nokrb-ipa

I opened a bug to make this easier in ONTAP with FreeIPA (1308667), so if you run into this, open a case and have it attached to the bug.

If you have questions, suggestions or get stuck (or if I got something wrong), comment below!

ONTAP 9.7 Feature Sneak Peek: NFS client to volume mapping

Since ONTAP was moved from 7-Mode to the more current clustered version, people have been asking for one specific feature:

The ability to map NFS clients to specific volumes and IP addresses.

We’ve had the ability to determine which clients are accessing IP addresses in a cluster (network connections active show), as well as CIFS/SMB session information (cifs session show), but could never get granular information about NFS.

Starting in ONTAP 9.7, we can.

Image result for homer woohoo

The use cases are varied, but usually fall into:

  • Need to discover who’s using a volume before performing migrations, cutovers, etc.
  • Troubleshooting issues
  • Load distribution

In my Unstructured NAS session at Insight 2019, I unveiled the new command that you can use to find NFS client to volume mapping:

cluster::> nfs connected-clients show ?
  [ -instance | -fields <fieldname>, ... ]
  [[-node] <nodename>]                                               Node Name
  [[-vserver] <vserver>]                                             Vserver
  [[-data-lif-ip] <IP Address>]                                      Data LIF IP Address
  [[-client-ip] <IP Address>]                                        Client IP Address
  [[-volume] <volume name>]                                          Volume Accessed
  [[-protocol] <Client Access Protocol>]                             Protocol Version
  [ -idle-time <[<integer>d][<integer>h][<integer>m][<integer>s]> ]  Idle Time (Sec)
  [ -local-reqs <integer> ]                                          Number of Local Reqs
  [ -remote-reqs <integer> ]                                         Number of Remote Reqs

This admin-level command will give you information on which clients are attached to which volume. Plus, if you’re using FlexGroup volumes, you get info all the way down to the member volume level.

For example:

cluster::> nfs connected-clients show -node node1 -vserver DEMO

Node: node1
Vserver: DEMO
Data-Ip: 10.193.67.219
Client-Ip Volume-Name Protocol Idle-Time Local-Reqs Remote-Reqs
--------------- ---------------- -------- ------------- ---------- -----------
10.62.1.173 Tech_ONTAP__0001 nfs3 13h 47m 22s 118 0
10.193.67.121 FGveeam2__0001 nfs3 23h 46m 2s 14326 0
10.193.67.121 FGveeam2__0002 nfs3 1d 8h 34m 0s 0 80118
10.193.67.121 FGveeam4__0001 nfs3 6h 34m 49s 15508 0
10.193.67.121 FGveeam4__0002 nfs3 23h 20m 39s 0 10246
10.193.67.121 FGveeam4__0003 nfs3 23h 20m 35s 10253 0
10.193.67.121 FGveeam4__0004 nfs3 23h 20m 38s 0 5139

(You perhaps notice the FlexGroup name above includes Veeam… that’s a story for another day. :-D)

Enjoy!

*ONTAP 9.7 is due out soon, so stay on the lookout for a blog post announcing its release!