Running PowerShell from Linux to Query SMB Shares in NetApp ONTAP

I recently got a question about how to perform the following scenario:

  • Run a script from Linux that calls PowerShell on a remote Windows client using Kerberos
  • Remote Windows client uses PowerShell to authenticate against an ONTAP SMB share

That’s some Inception-style IT work.

Inception Ending Explained: Christopher Nolan's Endless Spinning | Observer

The issue they were having was that the credentials used to connect to the Windows client weren’t passing through to the ONTAP system. As a result, they’d get “Access Denied” in their script when attempting to access the share. I figured out how to get this working and rather than let that knowledge rot in the far reaches of my brain, I’m writing this up, since in my Google hunt, I found lots of people had similar issues with Linux PowerShell (not necessarily to ONTAP).

This is a known issue with some workarounds listed here:

Making the second hop in PowerShell Remoting

One workaround is to use “Resource-based Kerberos constrained delegation,” where you basically tell the 3rd server to accept delegated credentials from the 2nd server via the PrincipalsAllowedToDelegateToAccount parameter in the ADComputer cmdlets. We’ll cover that in a bit, but first…

WAIT. I can run PowerShell on Linux???

Well, yes! And this article tells you how to install it:

Installing PowerShell on Linux

Now, the downside is that not all PowerShell modules are available from Linux (for example, ActiveDirectory isn’t currently available). But it works!

PS /> New-PSSession -ComputerName COMPUTER -Credential administrator@NTAP.LOCAL -Authentication Kerberos

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **

Id Name     Transport ComputerName ComputerType  State  ConfigurationName    Availability
-- ----     --------- ------------ ------------  ----- --------------------- ------------
9 Runspace9 WSMan     COMPUTER     RemoteMachine Opened Microsoft.PowerShell Available

In that document, they don’t list CentOS/RHEL 8, which can be problematic, as you might run into some issues with the SSL libraries (This blog calls one of those issues out, as well as a few others).

On my Centos8.3 box, I ran into this issue:

New-PSSession: This parameter set requires WSMan, and no supported WSMan client library was found. WSMan is either not installed or unavailable for this system.

Using the guidance from the blog listed earlier, I found that there were a couple of files not found:

# ldd /opt/microsoft/powershell/7/libmi.so
…
libssl.so.1.0.0 => not found
libcrypto.so.1.0.0 => not found
…

That blog lists 1.0.2 as what is needed and looks to be using a different Linux flavor. You can find the files you need/where they live with:

# find / -name 'libssl.so.1.'
/usr/lib64/.libssl.so.1.1.hmac
/usr/lib64/libssl.so.1.1
/usr/lib64/libssl.so.1.1.1g
/usr/lib64/.libssl.so.1.1.1g.hmac
/opt/microsoft/powershell/7/libssl.so.1.0.0

Then you can use the symlink workaround and those files show up properly with ldd:

ln -s libssl.so.1.1 libssl.so.1.0.0
ln -s libcrypto.so.1.1 libcrypto.so.1.0.0
ldd /opt/microsoft/powershell/7/libmi.so
...
libssl.so.1.0.0 => /lib64/libssl.so.1.0.0 (0x00007f41ce3fc000)
libcrypto.so.1.0.0 => /lib64/libcrypto.so.1.0.0 (0x00007f41cdf16000)
...

However, authenticating with the server also requires an additional step.

Authenticating Linux PowerShell with Windows

You can authenticate to Windows servers with Linux PowerShell using the following methods:

Basic Default Kerberos Credssp Digest Negotiate

Here’s how each auth method works (or doesn’t) without doing anything else.

BasicNew-PSSession: Basic authentication is not supported over HTTP on Unix.
DefaultNew-PSSession: MI_RESULT_ACCESS_DENIED
CredSSPNew-PSSession: MI_RESULT_ACCESS_DENIED
DigestNew-PSSession: MI_RESULT_ACCESS_DENIED
KerberosAuthorization failed Unspecified GSS failure.
NegotiateAuthorization failed Unspecified GSS failure.
Auth methods with Linux PowerShell and results with no additional configs

In several places, I’ve seen the recommendation to install gssntlmssp on the Linux client, which works fine for “Negotiate” methods:

PS /> New-PSSession -ComputerName SERVER -Credential administrator@NTAP.LOCAL -Authentication Negotiate

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **

Id Name Transport ComputerName ComputerType State ConfigurationName Availability
-- ---- --------- ------------ ------------ ----- ----------------- ------------
2 Runspace2 WSMan SERVER       RemoteMachine Opened Microsoft.PowerShell Available

But not for Kerberos:

PS /> New-PSSession -ComputerName SERVER -Credential administrator@NTAP.LOCAL -Authentication Kerberos

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **

New-PSSession: [SERVER] Connecting to remote server SERVER failed with the following error message : Authorization failed Unspecified GSS failure. Minor code may provide more information Configuration file does not specify default realm For more information, see the about_Remote_Troubleshooting Help topic.

The simplest way to get around this is to add the Linux client to the Active Directory domain. Then you can use Kerberos for authentication to the client (at least with a user that has the correct permissions, such as a domain administrator).

*Alternately, you could do all this manually, which I don’t recommend.

**For non-AD KDCs, config methods will vary.

# realm join NTAP.LOCAL
Password for Administrator:

# pwsh
PowerShell 7.1.3
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

PS /> New-PSSession -ComputerName SERVER -Credential administrator@NTAP.LOCAL -Authentication Kerberos

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********


 Id Name            Transport ComputerName    ComputerType    State         ConfigurationName     Availability
 -- ----            --------- ------------    ------------    -----         -----------------     ------------
  1 Runspace1       WSMan     SERVER          RemoteMachine   Opened        Microsoft.PowerShell     Available


So, now that we know we can establish a session to the Windows server where we want to leverage PowerShell, now what?

Double-hopping with Kerberos using Delegation

One way to use Kerberos across multiple servers (including NetApp ONTAP) is to leverage the PrincipalsAllowedToDelegateToAccount parameter.

The script I’ll use does a basic “Get-Content” call to a file in an SMB/CIFS share in ONTAP (similar to “cat” in Linux).

If I don’t set the PrincipalsAllowedToDelegateToAccount parameter, a credential passed from Linux PowerShell to a Windows server to ONTAP will use Kerberos -> NTLM (with a NULL user) for the authentication and this is the end result:

# pwsh test.ps1

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********

Test-Path: Access is denied
False
Get-Content: Access is denied
Get-Content: Cannot find path '\\DEMO\files\file-symlink.txt' because it does not exist.

In a packet capture, we can see the session setup uses NULL with NTLMSSP:

14    0.031496   x.x.x.x   x.x.x.y      SMB2 289  Session Setup Request, NTLMSSP_AUTH, User: \

And here’s what the ACCESS_DENIED looks like:

20    0.043026   x.x.x.x   x.x.x.y      SMB2 166  Tree Connect Request Tree: \\DEMO\files
21    0.043217   x.x.x.y   x.x.x.x      SMB2 131  Tree Connect Response, Error: STATUS_ACCESS_DENIED

To use Kerberos passthrough/delegation, I run this PowerShell command to set the parameter on the destination (ONTAP) CIFS server:

Set-ADComputer -Identity DEMO -PrincipalsAllowedToDelegateToAccount SERVER$

That allows the SMB session to ONTAP to set up using Kerberos auth:

2603 26.877660 x.x.x.x x.x.x.y SMB2 2179 Session Setup Request
2673 26.909735 x.x.x.y x.x.x.x SMB2 326 Session Setup Response
supportedMech: 1.2.840.48018.1.2.2 (MS KRB5 - Microsoft Kerberos 5)

And the tree connect succeeds (you may need to run klist purge on the Windows client):

2674 26.910117 x.x.x.x x.x.x.y SMB2 154 Tree Connect Request Tree: \demo\files
2675 26.910630 x.x.x.x x.x.x.y SMB2 138 Tree Connect Response

This is the result from the Linux client:

# pwsh test.ps1

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********

True
This is a file symlink.

So, how do we work around this issue if we can’t delegate Kerberos?

Using the NULL user and NTLM

Remember when I said the request without Kerberos delegation used the NULL user and NTLMSSP?

14    0.031496   x.x.x.x   x.x.x.y      SMB2 289  Session Setup Request, NTLMSSP_AUTH, User: \ 

The reason we saw “Access Denied” to the ONTAP CIFS/SMB share is because ONTAP disallows the NULL user by default. However, in ONTAP 9.0 and later, you can enable NULL user authentication, as described in this KB article:

How to grant access to NULL (Anonymous) user in Clustered Data ONTAP

Basically, it’s a simple two-step process:

  1. Create a name mapping rule for ANONYMOUS LOGON
  2. Set the Windows default NULL user in the CIFS options

Here’s how I did it in my SVM (address is the Windows client IP):

::*> vserver name-mapping create -vserver DEMO -direction win-unix -position 3 -pattern "ANONYMOUS LOGON" -replacement pcuser -address x.x.x.x/24

The Windows user needs to be a valid Windows user.

::*> cifs options modify -vserver DEMO -win-name-for-null-user NTAP\powershell-user

You can verify ONTAP can find it with:

::*> access-check authentication translate -node node1 -vserver DEMO -win-name NTAP\powershell-user
S-1-5-21-3552729481-4032800560-2279794651-1300

Once that’s done, we authenticate with NTLM and get access with the NULL user:

14 0.009012 x.x.x.x x.x.x.y SMB2 289 Session Setup Request, NTLMSSP_AUTH, User: \
27 0.075264 x.x.x.x x.x.x.y SMB2 166 Tree Connect Request Tree: \DEMO\files
28 0.075747 x.x.x.y x.x.x.x SMB2 138 Tree Connect Response

And the Linux client is able to run the PowerShell calls:

# pwsh test.ps1

PowerShell credential request
Enter your credentials.
Password for user administrator@NTAP.LOCAL: **********

True
This is a file symlink.

Questions? Comments? Add them below!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s