Behind the Scenes: Episode 240 – NetApp ONTAP S3

Welcome to the Episode 240, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

In this podcast episode,  NetApp StorageGRID has long been the go-to solution for S3 when dealing with the NetApp portfolio. But, due to popular demand, S3 is now in ONTAP as a public preview!

NetApp’s StorageGRID’s Senior Director and Object Store PM Duncan Moore (@NCdunc, dmoore@netapp.com) joins us, along with StorageGRID Solutions Architect Luke Mun (@lukamun, luke.mun@netapp.com), ONTAP S3 PM James Hunter (@jameshunter, james.hunter@netapp.com) and TME John Lantz (john.lantz@netapp.com), as we discuss the ins and outs of ONTAP S3 and where it fits alongside StorageGRID in the NetApp object storage solution portfolio.

Check out the ONTAP S3 TR here:

TR-4814: S3 Public Preview ONTAP 9.7

Note: LIST bucket was mentioned as unsupported in ONTAP 9.7; support for it has been added to 9.7P2.

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 240: NetApp ONTAP S3 – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Tech ONTAP Podcast · Episode 240 – NetApp ONTAP S3

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 239 – NetApp and Red Hat Summit 2020

Welcome to the Episode 239, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

In this podcast episode, Red Hat joins us to discuss this year’s virtual Red Hat Summit 2020, as well as the partnership with NetApp. And Sully the Monster makes a triumphant return!

sully-hearye

Joining us:

  • Nick Wallace, ISV Partner Alliance Manager, Red Hat (@nickalausw)
  • Andrew Sullivan, Principal Technical Marketing Manager, Cloud Platforms, Red Hat (@practicalAndrew)
  • Mark Cates, Principal Product Manager – HCI Solutions, NetApp (@cicurios)

For more information, see:

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 239: NetApp and Red Hat Summit 2020

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 238 – NetApp Zero Trust Solutions

Welcome to the Episode 238, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

In this podcast episode, NetApp security PM Juan Mojica (@juan_m_mojica) and TME Dan Tulledge (@dan_tulledge) join us to tell us about how NetApp helps enterprises architect a zero-trust security solution with ONTAP.

For more information, see:

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 238: NetApp Zero Trust – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 237 – Iguazio

Welcome to the Episode 237, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, Iguazio’s CTO Yaron Haviv (@yaronhaviv) and VP of Business Development Ori Lev Ran (oril@iguazio.com) join NetApp’s Sr. Solutions Manager of AI, Hoseb Dermanilian (@HosebDM) and myself to talk about Iguazio (@Iguazio) and where they fit in the AI/ML landscape.

To see a demo of Iguazio:

https://tv.netapp.com/detail/video/6149623013001

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 237: Iguazio – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 236 – How NVIDIA, NetApp and AI in Healthcare Can Help Fight the COVID-19 Pandemic

Welcome to the Episode 236, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast,  Dr. Mona Flores, (https://www.linkedin.com/in/monagflores/) the Global Lead for Hospitals and Clinical Partnerships from NVIDIA, joins NetApp’s Healthcare Business Development Manager Esteban Ruebens (@esteban_aihc) to discuss how NetApp and NVIDIA are teaming up to help healthcare professionals use AI, as well as how AI is being used to combat the COVID-19 pandemic.

Also, related this week, NetApp’s CEO George Kurian spoke about NetApp’s response to the pandemic on Fox Business:

https://video.foxbusiness.com/v/6146651560001/#sp=show-clips

For more information:

https://www.netapp.com/us/data-visionary/hannover-medical-school-fights-lung-disease-with-data.aspx

https://www.corescientific.com/news/core-scientific-offers-free-access-covid-19-research

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 236: How NVIDIA, NetApp and AI in Healthcare Can Help Fight the COVID-19 Pandemic – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 235 – NetApp MAX Data and Intel Whitepaper

Welcome to the Episode 235, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, we discuss the new collaborative white paper between NetApp and Intel on MAX Data. NetApp’s Mr. Memory Chris Gebhardt (@chrisgeb) and Intel’s Sridhar Kayathi (sridhar.r.kayathi@intel.com) join us to elaborate, as well as to myth-bust some of the thinking around what qualifies as storage class memory.

The new MAX Data white paper can be found here:

Maximize Database Performance with NetApp and Persistent Memory

https://www.intel.com/content/www/us/en/architecture-and-technology/optane-technology/netapp-max-data.html

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 235: NetApp MAX Data and Intel Whitepaper – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Mounting ONTAP CIFS/SMB shares with Linux – Guidelines and tips

Every now and then, the question of mounting CIFS/SMB shares in ONTAP using Linux clients comes up. Many people might consider this configuration to be a crime against nature.

garbodor

If you’re asking yourself “why would someone want to do this,” there are a few reasons.

  • Lock handling – SMB locks use oplocks. NFS locks use shared locks or delegations, depending on the NFS version and options being used. Generally this is fine, but in some cases, the style of lock might create complications with some applications. For example, Kafka on NFS has some issues with how NFS handles deletions when a file is locked:
    https://blog.usejournal.com/kafka-on-kubernetes-a-good-fit-95251da55837?gi=91867e828261
  • Legacy copiers/scanners – Many companies will use copiers/scanners to create image files that redirect to a NAS share in their environment. Many of those copiers use old SMB clients and are no longer under support, so they can’t be upgrades. And many of those scanners/copiers are expensive, so people aren’t interested in buying newer ones just to get newer SMB versions.
  • “How we’ve always done it” – This is a common reason that doesn’t always have technical logic associated with it, but you still have to roll with it.

There are other reasons out there as well, I’m sure. But this post will attempt to group the considerations, support statements and issues you might see in case someone else has that question.

Mounting SMB shares in ONTAP – General Guidelines

In 7-Mode ONTAP, mounting SMB shares with clients like AIX had no issues. This was because 7-Mode supported a lot of old SMB technology concepts that newer versions of ONTAP have deprecated. Even though SMB shares on Linux worked in 7-Mode, there was never an official supported configuration for it.

NetApp ONTAP has traditionally only supported Apple OS and Windows OS for CIFS/SMB workloads.

You can find the supported clients in the Interoperability Matrix here:

https://mysupport.netapp.com/matrix/

With newer versions of ONTAP, you *can* get SMB/CIFS working properly, provided you take the following into account:

  • CIFS/SMB clients should support UNICODE for CIFS/SMB communication. Clustered ONTAP doesn’t support non-UNICODE in any release and never will. Newer CIFS/SMB clients should support UNICODE. AIX is a good example of an older Linux client that has some challenges using SMB because it still uses non-UNICODE in some client installs.
  • CIFS/SMB versions mounting should reflect the supported SMB versions in ONTAP. ONTAP has deprecated SMB1, so you’d need to mount using SMB2 or SMB3.
  • CIFS/SMB enabled features in ONTAP should reflect the feature support for the CIFS/SMB Linux clients. For example, my Centos7 Linux Samba client doesn’t support large MTU, SMB3 encryption, multichannel, etc. That can cause failures in the mounts (for an example, see below).
  • If using Kerberos for the CIFS/SMB mounts, be sure the CIFS/SMB ONTAP server has a valid SPN for the CIFS service. The name should match the mount name. For example, if your CIFS/SMB server is named “CIFS” then the SPN would be cifs/cifs.domain.com.

Common Issues when Mounting CIFS/SMB

One of the main issues people run into with SMB mounts in Linux (provided they follow the guidelines above when using ONTAP) is using the wrong command syntax. For example, when you specify the share path, you use either \\\\name\\share (extra slashes to tell Linux that the \ isn’t an operand) or you use //name/share.

The following NetApp KB article does a good job of giving command examples for the right way to mount SMB:

How to access CIFS from Linux machine using SAMBA

Usually the command will tell you if the wrong option is being used, but sometimes the errors are less than helpful. These are a few errors I ran into.

Incorrect Mount Option

This was pretty self-explanatory. First, I didn’t have the cifs-utils installed. Second, I used the wrong command syntax.

Mount point does not exist

Again, self explanatory. The directory you’re trying to mount to needs to exist.

No such file or directory (when mounting)

This means you specified the wrong export path on the NFS server. Check your junction-paths and try again.

Host is down

This one was a tricky error, as it suggests that the server was not up. In some cases, that may truly be the issue. But it wasn’t in my case.

# mount -t cifs -o user=cifsuser \\\\DEMO\\nas /mnt/nas
Password for cifsuser@\DEMO\nas: **********
mount error(112): Host is down

But, in reality, the issue was that I didn’t specify the SMB version and it tried to use SMBv1.0 by default.

Specifying the SMB version (-o vers=3.0) got past that issue.

Required key not available

This is a Kerberos specific error. In my case, the cifs/servername.domain.com SPN didn’t exist for the hostname I used in the UNC path. You can see that in a packet capture.

Permission denied

This error is fairly useless in both Windows and Linux in a lot of cases – mainly because it can mean a variety of things. Sometimes, it really is an access issue (such as share or file-level permissions). But in my testing, I also ran into this issue when I had unsupported SMB features enabled on my CIFS server in ONTAP.

# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas
Password for administrator@//DEMO/nas: **********
mount error(13): Permission denied

In a packet trace, I could see the client telling me what it supported:

smb-capabilities

But the reply didn’t really mention the issue. So I made sure to disable the following CIFS/SMB features:

  • SMB3 encryption (cifs security modify)
  • Large MTU and SMB Multichannel (cifs options modify)

Once I did that, I was able to mount.

[root@centos7 /]# mount -t cifs -o vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas
Password for administrator@//DEMO/nas: **********
[root@centos7 /]# touch /mnt/nas/smbfile
[root@centos7 /]# ls -la /mnt/nas
total 1
drwxr-xr-x 2 root root 0 Mar 27 14:36 .
drwxr-xr-x. 9 root root 97 Mar 27 10:17 ..
-rwxr-xr-x 1 root root 0 Mar 27 14:36 smbfile

For Kerberos, you would just specify sec=krb5.

# mount -t cifs -o sec=krb5,vers=3.0,user=administrator,domain=NTAP.LOCAL //DEMO/nas /mnt/nas

There you go! Hopefully this helped someone from going down a Google K-hole. Leave comments or suggestions below.

Behind the Scenes: Episode 234 – NetApp Keystone

Welcome to the Episode 234, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week on the podcast, we discuss NetApp Keystone with Sunitha Rao (Sunitha.rao@netapp.com, @sira_10) and Jean Banko (@bankonAI) and how it is simplifying cloud consumption.

Podcast Transcriptions

We also are piloting a new transcription service, so if you want a written copy of the episode, check it out here (just set expectations accordingly):

Episode 234: NetApp Keystone – Podcast Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Getting ONTAP NFS Kerberos to Work with FreeIPA

Image result for hamburglar

Obviously, with the social distancing/lockdowns happening, I have had more time to write up blogs on things. So, here’s another one. If you have suggestions for topics, let me know and I’ll see about writing them up.

Before we get to the meat of what the title is (or scroll down to the bottom if you want the quick and dirty steps), let’s recap NFS security and Kerberos…

NFS is a protocol that allows multiple clients communicate to a single storage endpoint as a way to share data across a network. Because data is transmitted over a network, it’s important to be able to secure that data, both at-rest and in-flight.

At-rest Security in NFS

At-rest security refers to security applied to data residing on the storage system, as well as the interaction between NFS client and NFS server to negotiate things like NFS version, identity, etc.

At-rest security for NFS includes:

  • Export policies and rules
  • Permissions/ACLs
  • User/group owners
  • Name ID strings (NFSv4.0 and later)

However, when data passes over the wire, packet sniffers are able to see identifying information like IP addresses, users, groups, permissions, file names, paths, etc. All it would take is someone being able to see this information to easily create duplicate set ups to “spoof” users and groups and gain access to data. At-rest security mostly protects you from threats that don’t have this knowledge or expertise. To prevent bad actors from getting information from the transmission of data, you’d need to set up in-flight security.

In-flight Security in NFS

In-flight security refers to securing the actual data packets in transit.

For NFS, this can be done in a few ways:

For ONTAP, tunneling NFS over SSH or stunnel isn’t supported, but you can use NFS Kerberos.

For a deeper look at NFS Kerberos in ONTAP, see:

NFS Kerberos in ONTAP Primer

Kerberos Security Flavors

NFS Kerberos has 3 methods that can be used to secure things. Each provides an additional level of security, but also adds extra performance overhead.

Keep in mind that NFSv3 *can* use Kerberos, but only the NFS portion of the protocol will use it. Ancillary protocols like mountd, portmapper, NLM, etc. will still be unencrypted. The most secure version of NFS available is NFS v4.x, which combines all the NFS ancillary protocols into a single port and compound NFS calls, which can all be Kerberized.

krb5 – Encrypts the authentication only

Remember when I said if a bad actor stole information about the client, user, etc. that they could spoof that user to get access? Well, with krb5, that becomes harder to do. You can have all the information about the NFS export, but to gain access to a Kerberized mount, you’d need to have a valid username and password to get a Kerberos TGT, which would then be used to get the NFS service ticket. In addition, your client would also need to have a Kerberos ticket and keytab to gain access, so unless your attacker has KDC admin rights, they won’t be able to access the mount.

krb5i – Authentication with data checksums/integrity checking

Krb5 secures the initial authentication for NFS, but the actual NFS data packets are still transmitted in clear text across the wire. If a bad actor were to plug in and sniff those data packets in flight, they could see everything. In addition, bad actors could also act as a “man in the middle” to intercept packets and then transmit their own data if they chose. Krb5i prevents this by using checksums on the client and server to verify that all the data arriving and leaving is coming from a trusted source. The data is still visible, but it can’t be interfered with.

krb5p – Authentication/integrity checking/end to end encryption (privacy)

For the ultimate in-flight security hammer for NFS, you would use krb5p. This combines what krb5i does with in-flight encryption of all NFS packets. That means all NFS packets will be encrypted with the enctype specified in the Kerberos configuration. This can add considerable overhead for NFS performance, but will also prevent NFS packets from being sniffed easily.

How to set up NFS Kerberos in ONTAP

Setting up Kerberos in ONTAP requires a few things:

  • A Key Distribution Center (Microsoft AD, MIT Kerberos, FreeIPA, RedHat IDM)
  • DNS server or static host entries
  • Optional (but recommended): LDAP server for UNIX identity management
  • Configuring NFS clients and ONTAP Service Principal Names

The KDC is the central hub for Kerberos operations and is responsible for handing out Kerberos tickets to clients, users and services for authentication in a Kerberos realm.

DNS is used to resolve IP addresses to host names, which are then passed to the KDC as SPN requests. For example, if client 10.10.10.10 tries to get a Kerberos ticket, a DNS reverse lookup will be done to find the hostname FQDN “client1.domain.com” Then a SPN request for host/client1.domain.com@DOMAIN.COM is made to the KDC. If the SPN exists and matches the request, then the Kerberos request moves on to the next steps. If it doesn’t exist, the request fails.

LDAP is used to centralize the UNIX identities for users and groups to ensure clients and servers have the same information all the time, without manual intervention. This is less important to Kerberos and more important to NFSv4.x and NFS permission resolution.

NFS client configuration for Kerberos on newer clients is fairly straightforward; you configure DNS (so it can find the Kerberos realm name and client hostname) and then simply use one of the automated tools to “join” the Kerberos realm. This automatically creates the service principal and transfers the keytab files. Where it gets tricky is if you have to do a manual Kerberos configuration. TR-4073 and TR-4616 can be of some guidance there, as well as a bevy of articles across the web.

NFS server/ONTAP configuration for Kerberos is relatively simple; you configure DNS and the Kerberos realm and then you enable Kerberos on a network interface. When you do that, ONTAP interacts with the KDC you specified in the Kerberos realm and automates the principal creation – provided the KDC is using standard Kerberos interaction, such as Microsoft Active Directory or kadmin for Linux.

Now we’re getting to the part the title hinted about… FreeIPA Kerberos setup.

Why do I need a blog on FreeIPA Kerberos with ONTAP? You just said it was easy!

I did say it was easy – if the KDC is using standard kadmin. However, FreeIPA happens to use a wrapper over kadmin for KDC management and discourages the use of kadmin for management of service principals. In fact, by default, they lock things down pretty tight.

In FreeIPA, there is a GUI to make things simpler to use. There are also “ipa” commands to interact via the CLI, such as ipa user-add or ipa service-add. In most Linux KDCs, there was kadmin to manage things. In fact, there are *two* kadmins. One is for local management (kadmin.local) and the other is for remote management (kadmin). For more info on the differences, see:

https://docs.oracle.com/cd/E19683-01/816-0211/6m6nc66tj/index.html

The remote kadmin is what ONTAP interacts with for Linux KDCs. When you use the automated ONTAP method to add the NFS service principal, the following happens:

  • A prompt for a username and password to issue a kinit to the KDC to gain access to kadmin
  • get_principal is run via remote kadmin to see if the SPN exists
    • If the SPN doesn’t exist, create_principal is run via remote kadmin
    • If the SPN does exist, modify_principal is run via remote kadmin

All of these kadmin commands require the proper privileges to run successfully. In FreeIPA, remote kadmin is locked down by default to even run get_principal.

For example:

# ipa user-add krbadmin

---------------------
Added user "krbadmin"
---------------------
User login: krbadmin
First name: Kerberos
Last name: Admin
Full name: Kerberos Admin
Display name: Kerberos Admin
Initials: KA
Home directory: /home/krbadmin
GECOS: Kerberos Admin
Login shell: /bin/sh
Principal name: krbadmin@CENTOS-LDAP.LOCAL
Principal alias: krbadmin@CENTOS-LDAP.LOCAL
Email address: krbadmin@centos-ldap.local
UID: 1971600007
GID: 1971600007
Password: False
Member of groups: ipausers
Kerberos keys available: False

# kadmin -p krbadmin
Authenticating as principal krbadmin with password.
Password for krbadmin@CENTOS-LDAP.LOCAL:
kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Operation requires ``get'' privilege while retrieving "nfs/server.domain.com@DOMAIN.COM".

The ONTAP error we’d see is less clear, however. The error suggests that the SPN *already exists*.

Error: NFS Kerberos bind SPN procedure failed
  [  0 ms] Creating account in Unix KDC
  [    30] Successfully connected to ip 10.x.x.x, port 749
           using TCP
**[    40] FAILURE: Unexpected state: Error 1142 at
**         file:src/utils/secd_kadmin_utils.cpp
**         func:createVifKrbAccountUsingKadmin line:227
**[    40] FAILURE: spn already exists. Failed to reuse spn
**         'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
**         admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
**         Unknown code 0
  [    41] Uncaptured failure while creating account

Obviously, the error should refer to some permissions issue. When we see this error in ONTAP,  we can verify on the KDC what is happening via the /var/log/kadmind.log file. In this case, we see “unauthorized request.”

Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Request: kadm5_init, kadmin/admin@CENTOS-LDAP.LOCAL, success, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x, vers=2, flavor=6
Mar 20 16:45:03 centos8-ipa.centos-ldap.local kadmind[2929](Notice): Unauthorized request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, client=kadmin/admin@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

To give permissions to users to use remote kadmin, you have to modify the /var/kerberos/krb5kdc/kadm5.acl file.

In my setup, this is how I configured the ACL:

# cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@CENTOS-LDAP.LOCAL *
ontap@CENTOS-LDAP.LOCAL *
krbadmin@CENTOS-LDAP.LOCAL *

Then you restart IPA services:

# service ipa restart
Redirecting to /bin/systemctl restart ipa.service

Now, my user can query the principal:

kadmin: get_principal nfs/server.domain.com@DOMAIN.COM
get_principal: Principal does not exist while retrieving "nfs/server.domain.com@DOMAIN.COM".

However, ONTAP gets a new (less descriptive) error when trying to interact with the KDC:

Error: NFS Kerberos bind SPN procedure failed
[ 1 ms] Creating account in Unix KDC
[ 29] Successfully connected to ip x.x.x.x, port 749
using TCP
**[ 45] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:createVifKrbAccountUsingKadmin line:223
**[ 45] FAILURE: Failed to create spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'kadmin/admin@CENTOS-LDAP.LOCAL', error:
** Invalid argument
[ 45] Uncaptured failure while creating account

The /var/log/kadmind.log is also fairly useless here:

Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_get_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Principal does not exist, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x
Mar 23 12:17:58 centos8-ipa.centos-ldap.local kadmind[14965](Notice): Request: kadm5_create_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid argument, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

ONTAP is just reporting what kadmin says!

Our clue comes from when we try to manually create the SPN using kadmin:

kadmin: add_principal -randkey nfs/server.domain.com@DOMAIN.COM
WARNING: no policy specified for nfs/server.domain.com@DOMAIN.COM; defaulting to no policy
add_principal: Kerberos database constraints violated while creating "nfs/server.domain.com@DOMAIN.COM".

Same goes for modify_principal:

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
(vserver nfs kerberos interface enable)

Username: ontap

Password:

Warning: An account that matches the given service principal name already exists. Re-using this account deletes and re-creates the account if it is not shared by the LIFs in Vserver "NFS". This
invalidates any existing Kerberos service tickets and keys for this account. Do you want to re-use this account? {y|n}: y

Error: NFS Kerberos bind SPN procedure failed
[ 0 ms] Creating account in Unix KDC
[ 39] Successfully connected to ip x.x.x.x, port 749
using TCP
[ 45] Re-using the account for
spn=nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
**[ 53] FAILURE: Unexpected state: Error 1142 at
** file:src/utils/secd_kadmin_utils.cpp
** func:randomizePasswordUsingKadmin line:287
**[ 53] FAILURE: Failed to randomize password for spn
** 'nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL' using
** admin spn 'ontap@CENTOS-LDAP.LOCAL'
[ 54] Uncaptured failure while creating account


Mar 24 11:14:54 centos8-ipa.centos-ldap.local kadmind[17208](Notice): Request: kadm5_randkey_principal, nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL, Invalid key/salt tuples, client=ontap@CENTOS-LDAP.LOCAL, service=kadmin/admin@CENTOS-LDAP.LOCAL, addr=x.x.x.x

And delete_principal:

::*> kerberos interface disable -vserver NFS -lif ipa-krb
(vserver nfs kerberos interface disable)

Username: ontap

Password:

Warning: This command deletes the service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" from the machine account on the KDC. Do you want to continue? {y|n}: y

Error: command failed: Failed to disable NFS Kerberos on LIF "ipa-krb". Failed to delete the account associated with the Kerberos service principal name. Reason: cifs smb kadmin error.

So that means, even though we gave full rights to that user in the kadm5.acl file, we can’t create principals in FreeIPA using kadmin. They want to funnel you to use the IPA tools – likely for a good reason. There’s probably a bunch of other automated steps that take place with those tools, so this is in the interest of simplicity. and security.

How do we get past this?

There are two options here.

  1. Open up the permissions on FreeIPA to use kadmin to create/add/modify/delete principals. (I haven’t figured out how to do this yet)
  2. Manually create the SPN and keytab file and then use ONTAP to transfer the keytab via URI

I went with option 2.

The TL;DR steps, as provided by Alexander Bokovoy in the comments:

- kinit admin
- ipa host-add demo-ipa.centos-ldap.local
- ipa service-add nfs/demo-ipa.centos-ldap.local
- ipa-getkeytab -p nfs/demo-ipa.centos-ldap.local -k ./nfs.keytab -e aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

Remember that ONTAP only supports the following enctypes for NFS Kerberos:

  • AES-128 and AES-256
  • DES and 3DES

Then, you copy that file to your HTTP or FTP server. The address to the file will be used in the ONTAP CLI command.

Note: If you’re using IIS, you may have to allow .keytab as a MIME type.

For example:

iis-mime-keytab

Once the file is web-accessible, you run the kerberos interface enable command and use the -keytab-uri option to upload the keytab. Unsupported enctypes will get discarded.

::*> kerberos interface enable -vserver NFS -lif ipa-krb -spn nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL -keytab-uri http://web-server/files/ipakrb-ontap.keytab
(vserver nfs kerberos interface enable)

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

Warning: Keys for encryption types "des-cbc-crc,des3-cbc-sha1,aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96" are required for Vserver "NFS" but found keys only for encryption types
"aes128-cts-hmac-sha1-96,aes256-cts-hmac-sha1-96". Keys for encryption types "des-cbc-crc,des3-cbc-sha1" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL" are
missing. Available keys will be imported. Do you want to continue? {y|n}: y

Warning: Skipping unsupported encryption type "25" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".
Warning: Skipping unsupported encryption type "26" for service principal name "nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL".

You’ll also want to ensure you have a valid UNIX user or name mapping rule for the krb-spn name mapping for the NFS clients. That’s covered in TR-4616 in detail.

Once that’s all done, NFS Kerberos mounts should work fine!

[root@centos8-1 sysconfig]# mount -o nfsvers=4,minorversion=1,sec=krb5 x.x.x.x:/kerberos /mnt
[root@centos8-1 sysconfig]# mount | grep krb
x.x.x.x:/kerberos on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=krb5,clientaddr=x.x.x.y,local_lock=none,addr=x.x.x.x)
# su ipa-user
sh-4.4$ cd /mnt
sh: cd: /mnt: Permission denied
sh-4.4$ kinit
Password for ipa-user@CENTOS-LDAP.LOCAL:
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ cd /mnt
sh-4.4$ klist -e
Ticket cache: KCM:1971600003
Default principal: ipa-user@CENTOS-LDAP.LOCAL

Valid starting Expires Service principal
03/23/2020 14:39:30 03/24/2020 14:39:27 krbtgt/CENTOS-LDAP.LOCAL@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
03/23/2020 14:40:16 03/24/2020 14:39:27 nfs/demo-ipa.centos-ldap.local@CENTOS-LDAP.LOCAL
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.4$ touch krb5file
sh-4.4$ ls -la
total 4
drwxrwxrwx. 2 root root 4096 Mar 23 14:40 .
dr-xr-xr-x. 17 root root 224 Jan 21 10:03 ..
-rw-------. 1 ipa-user admins 0 Mar 23 14:40 krb5file
-rw-r--r--. 1 root root 0 Mar 23 14:15 v3nokrb
-rw-r--r--. 1 root root 0 Mar 23 14:16 v4nokrb
-rw-------. 1 ipa-user admins 0 Mar 23 14:22 v4nokrb-ipa

I opened a bug to make this easier in ONTAP with FreeIPA (1308667), so if you run into this, open a case and have it attached to the bug.

If you have questions, suggestions or get stuck (or if I got something wrong), comment below!

Using Windows Lightweight Directory Services for UNIX Identity Management with ONTAP

Windows Active Directory domains have been the way to leverage UNIX identity management in environments using Windows, given the tight integration with Kerberos, Windows accounts and ease of use. I cover a lot of this in TR-4073 (with a new LDAP-only TR coming out soon).

But, it doesn’t always fit all use cases.

For example, what if:

  • You don’t have a Windows Active Directory domain?
  • You don’t have access or permission to modify Active Directory domain accounts to use UNIX attributes?
  • You don’t need a full-fledged AD domain with hundreds or thousands of users/groups, but only need a handful of users and groups for a single application?

There are likely other use cases that could apply here, but if you need LDAP without dealing with AD domains, we can leverage something called Windows Active Directory Lightweight Directory Services for identity management and LDAP services.

Image result for windows active directory lds

This also illustrates that pretty much *any* LDAP server can be used with ONTAP.

For example, this is a query from ONTAP to a server running AD LDS:

::*> getxxbyy getgrlist -node node1 -vserver NFS -username lds
(vserver services name-service getxxbyyy getgrlist)
pw_name: lds
Groups: 1101 1102


::*> getxxbyyy getpwbyname -node node1 -vserver NFS -username lds
(vserver services name-service getxxbyyy getpwbyname)
pw_name: lds
pw_passwd:
pw_uid: 1001
pw_gid: 1101
pw_gecos:
pw_dir: 
pw_shell:

The configuration from ONTAP is identical for any LDAP server. In my case, I used the read-only LDAP client schema template named “MS-AD-BIS.”

Basic configuration steps:

  • Create the LDAP client (“ldap client create” using the stock LDAP template MS-AD-BIS)
  • Create/enable LDAP on the SVM (ldap create)
  • Configure DNS on the SVM (dns create/modify)
  • Modify ns-switch to use LDAP (ns-switch modify)

This was my LDAP client configuration:

::*> ldap client show -client-config LDS

Vserver: NFS
Client Configuration Name: LDS
LDAP Server List: PARISI-WIN2019
(DEPRECATED)-LDAP Server List: -
Active Directory Domain: -
Preferred Active Directory Servers: -
Bind Using the Vserver's CIFS Credentials: false
Schema Template: MS-AD-BIS
LDAP Server Port: 389
Query Timeout (sec): 3
Minimum Bind Authentication Level: anonymous
Bind DN (User): administrator
Base DN: CN=LDS-LDAP,DC=PARISI-WIN2019
Base Search Scope: subtree
User DN: -
User Search Scope: subtree
Group DN: -
Group Search Scope: subtree
Netgroup DN: -
Netgroup Search Scope: subtree
Vserver Owns Configuration: false
Use start-tls Over LDAP Connections: false
Enable Netgroup-By-Host Lookup: false
Netgroup-By-Host DN: -
Netgroup-By-Host Scope: subtree
Client Session Security: none
LDAP Referral Chasing: false
Group Membership Filter:

The hardest part of this configuration for me was the AD LDS side, as there are a bunch of manual steps involved.

Configuring AD LDS for LDAP Services

The best guide I came across was this one:

AD LDS Identity Mapping for Services for NFS

Even with this guide, however, there were a few tweaks that needed to be made to the steps. For example, in the guide, they used the AD Schema Manager. But since this isn’t a full AD instance, you might be stuck using ADSI Edit.

One place I also got stuck was a RTFM moment where I forgot in my initial configuration to create an application partition. This is essential, as this is where you will be creating the OUs/Containers, users and groups for the LDAP services.

When creating the new objects, you’ll want to consider populating the following fields in the users/groups for use with ONTAP (based on the MS-AD-BIS default schema).

Already populated at creation

When you create a new object, “cn” gets autopopulated. This is used to pull the Group names. (User names use “uid” by default)

  • cn

Required

“Required” here means that ONTAP won’t be able to query for the object without these attributes populated properly.

  • uid
  • uidNumber
  • gidNumber

Optional, but recommended

These attributes are recommended, but things will mostly work without setting them (unless you require proper group memberships).

  • unixHomeDirectory
  • memberUid or member (for group memberships)

Optional for specific use cases

This is basically only if you need things like netgroup services or name mapping to Windows users.

  • sAMAccountName (for asymmetric name mappings to Windows)
  • nisObject
  • nisMapName
  • nisMapEntry
  • nisNetgroupTriple
  • memberNisNetgroup
  • loginShell (would need to be added manually)

User and Group creation

For user and group management/creation, you can either use ADSI Edit or PowerShell cmdlets. The ADSI Edit method is covered in the linked guide above.

For PowerShell you can use the standard Get-ADuser/Group and New-ADUser/Group. The catch is that you may have to specify the server – especially if your LDS server is a member of the AD domain.

For example:

PS C:\> get-aduser -Identity lds -server PARISI-WIN2019 -Properties uid,uidNumber,gidNumber,gecos,unixHomeDirectory

DistinguishedName : CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
Enabled : False
gecos : AD LDS User
gidNumber : 1101
GivenName :
Name : lds
ObjectClass : user
ObjectGUID : 087692ff-ae7f-4171-922a-98accfdfdaa8
SID : S-1-396492173-265181619-2703889971-1168526272-875569285-466579950
Surname :
uid : {lds}
uidNumber : 1001
unixHomeDirectory : /u/lds
UserPrincipalName : lds@NTAP.LOCAL

PS C:\> Get-ADGroup -Identity ldsgroup1 -server PARISI-WIN2019 -Properties gidNumber,member,memberUid

DistinguishedName : CN=ldsgroup1,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
gidNumber         : 1101
GroupCategory     : Security
GroupScope        : Global
member            : {CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019}
Name              : ldsgroup1
ObjectClass       : group
ObjectGUID        : 5764424e-aba4-4e68-bff5-b7a6989d3d0c
SID               : S-1-396492173-265181619-2048905208-1111646721-233999250-1998119334

PS C:\> Get-ADGroup -Identity ldsgroup2 -server PARISI-WIN2019 -Properties gidNumber,member,memberUid

DistinguishedName : CN=ldsgroup2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
gidNumber         : 1102
GroupCategory     : Security
GroupScope        : Global
member            : {CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019}
Name              : ldsgroup2
ObjectClass       : group
ObjectGUID        : 8aec090c-0865-4b80-bd4a-723884524ff6
SID               : S-1-396492173-265181619-3623710820-1184275431-3423871139-2897951880

Note that in the group examples, I used “member” – this enables ONTAP to use RFC-2307bis. When you add members using “member,” you use the “Add DN” button in ADSI Edit or the Add-ADGroupMember PowerShell cmdlet. Then you use the full DN of the user.

For example:

PS C:\> Add-ADGroupMember -Identity ldsgroup2 -Members "CN=lds2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019" -Server PARISI-WIN2019

PS C:\> Get-ADGroup -Identity ldsgroup2 -server PARISI-WIN2019 -Properties member

DistinguishedName : CN=ldsgroup2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019
GroupCategory : Security
GroupScope : Global
member : {CN=lds2,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019,
CN=lds,CN=Accounts,CN=LDS-LDAP,DC=PARISI-WIN2019}
Name : ldsgroup2
ObjectClass : group
ObjectGUID : 8aec090c-0865-4b80-bd4a-723884524ff6
SID : S-1-396492173-265181619-3623710820-1184275431-3423871139-2897951880

Now ONTAP will be able to query for both groups for that user:

::*> getxxbyy getgrlist -node node1 -vserver NFS -username lds2
(vserver services name-service getxxbyyy getgrlist)
pw_name: lds2
Groups: 1101 1102

So there it is – ONTAP using AD LDS for UNIX Identity Management.