TECH::Feeling insecure about NFS?

There’s an old joke about NFS…

not-for-security

And, in many cases, it’s true. NFS versions prior to NFSv4.x did not do much for security. This is true for any client or server.

Some of the issues include:

  • Multiple well-known ports for NFSv3 and prior (NLM, portmapper, mountd, NFS, NSM, rquota)
  • Plain text UID/GID and host information over the wire
  • Kerberos not as useful in NFSv3 (for instance, you cannot Kerberize NLM)

In this blog, I will illustrate just how easy it is to gain access to an exported filesystem with just a little bit of info gathered from a packet capture. I’ll be focusing on clustered Data ONTAP, since that’s what the cool kids are using these days, but these principals can be applied to any NFS implementation, as they adhere to RFC standards.

How NFS access is obtained

NFS access is controlled by two main gateways.

  • Export policies/rules
  • UNIX permissions

Export policies/rules

Export policies/rules are similar to CIFS shares. These simply present the data object to a client for network access. Export policies and rules are covered in great detail in TR-4067: NFS Best Practices and Implementation Guide. These control access based on the client’s IP or hostname rather than by a user, so all one would need to gain access to an export would be an IP address. If I gained access to a network, I could easily change my IP address and hostname to what I see in a trace to “become” that host.

Below is a sample trace where I get exactly that information:

mount-client-info

So now I know the client hostname, the client IP, the NFS server IP and the exported path. Yikes!

In the reply packet, I can see the allowed access, so I know exactly what AUTH flavors I have available to me. Once I see AUTH_UNIX (aka AUTH_SYS), I know I’m in business.

mount-client-info-reply

With AUTH_SYS, I know I don’t need a Kerberos ticket to get in to the volume. All I need is the client that has root access to do some real damage.

In this case, my export policy rule is set to be pretty wide open. In this case, I am not practicing good security. I just want ease of access. Below is the sample of the policy rule (on clustered Data ONTAP):

cluster::*> export-policy rule show -vserver SVM -policyname wideopen -instance
(vserver export-policy rule show)

Vserver: SVM
Policy Name: wideopen
Rule Index: 1
Access Protocol: any
Client Match Hostname, IP Address, Netgroup, or Domain: 0.0.0.0/0
RO Access Rule: any
RW Access Rule: any
User ID To Which Anonymous Users Are Mapped: 0
Superuser Security Types: any
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: unrestricted
Vserver Change Ownership Mode: unrestricted

So with the above policy and NFSv3, all anyone would need is to know *a* client IP address in my network. It’s much better to limit the client match to a hostname, IP or netgroup to ensure that someone attempting access would at least need to make the effort to sniff packets. I may also want to specify an AUTH type, preferably Kerberos (we’ll touch on that later). I’d also be much better off by not setting my “superuser” rule to “any” which allows carte blanche root access to anyone logging in as root on a client. Oh, and definitely don’t want “anon” to be 0. This one is much better:

cluster::*> export-policy rule show -vserver SVM -policyname map-anon -instance
(vserver export-policy rule show)

Vserver: SVM
Policy Name: map-anon
Rule Index: 1
Access Protocol: any
Client Match Hostname, IP Address, Netgroup, or Domain: 10.228.225.140
RO Access Rule: any
RW Access Rule: any
User ID To Which Anonymous Users Are Mapped: test
Superuser Security Types: none
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: unrestricted

Vserver: SVM
Policy Name: map-anon
Rule Index: 2
Access Protocol: any
Client Match Hostname, IP Address, Netgroup, or Domain: 10.228.225.128
RO Access Rule: none
RW Access Rule: none
User ID To Which Anonymous Users Are Mapped: test
Superuser Security Types: none
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: unrestricted
2 entries were displayed.

Now I have allowed access to the volume to only two IP addresses. I’ve also set these clients to squash root to the anon user, which is not UID 0. So, much more secure, but we can do better.

UNIX Permissions

The next level of security for NFS after exports would be UNIX permissions. These are the file/directory level access rights for users and groups.

In NFSv3, these are very limited. You essentially get owner, group and everyone else. That means you can’t assign multiple groups to a file or folder unless you nest the groups. And multiple owners?

Fuggedaboutit.

photo_james_gandolfini-500x373

Additionally, with NFSv3, we face the same issue we faced with hostnames over the wire – I can see what the file permissions are and what UID/GID I would need to gain access.

In the trace excerpt below, I can see in a LOOKUP reply, my file has 755 permissions and the owner/group is 0:0. So all I need to do is become root. Good thing I locked down root in my export policy above!

nfs-access

But I still have “5” (or read) access to everyone else. So all I need to do is become any old user to gain access to read this file. Maybe this file has the salaries of everyone at my company. Maybe it has the secret recipe for Coca Cola. Or maybe you have Excel files with passwords

……………………………………..________
………………………………,.-‘”……………….“~.,
………………………..,.-“……………………………..”-.,
…………………….,/………………………………………..”:,
…………………,?………………………………………………,
………………./…………………………………………………..,}
……………../………………………………………………,:`^`..}
……………/……………………………………………,:”………/
…………..?…..__…………………………………..:`………../
…………./__.(…..”~-,_…………………………,:`………./
………../(_….”~,_……..”~,_………………..,:`…….._/
……….{.._$;_……”=,_…….”-,_…….,.-~-,},.~”;/….}
………..((…..*~_…….”=-._……”;,,./`…./”…………../
…,,,___.`~,……”~.,………………..`…..}…………../
…………(….`=-,,…….`……………………(……;_,,-”
…………/.`~,……`-………………………………./
………….`~.*-,……………………………….|,./…..,__
,,_……….}.>-._……………………………..|…………..`=~-,
…..`=~-,__……`,……………………………
……………….`=~-,,.,………………………….
…………………………..`:,,………………………`…………..__
……………………………….`=-,……………….,%`>–==“
…………………………………._……….._,-%…….`
……………………………..,

Who knows, but it sure would be easy to find out at this point!

Security Lockdown: NFSv4 with Kerberos

There are two main ways to lock down NFS outside of export policies and mode bit permissions: NFSv4 and Kerberos. NFSv4 provides the following security benefits over NFSv3:

  • Single port used for NFS (all other protocol pieces live on this same port); less ports to open on firewalls
  • Identity domain strings to reduce UID/GID visibility and to enforce domain ID mapping for proper authentication
  • Better integration with Kerberos/AUTH_GSS
  • NFSV4 ACLs – more granularity for permissions than mode bits

NOTE: NFSv4.x is covered in more detail in TRs 3580, 4067 and 4073.

In the following packet capture, when I look for information about the UID/GID, I see the following:

nfs4-access

In the above, the permissions are still 755. And I see that the owner is root, but I also see that root has to belong to the NFSv4 ID domain of domain.win2k8.netapp.com. I can still spoof this user (especially when it’s root) by modifying the client to leverage NFSv4 ID domains, etc. But for users that I do not have the UID/GID for, it’s much harder.

Consider this scenario… I have a file called “nfsv4file.” It is owned by test@domain.win2k8.netapp.com.

# ls -la | grep nfsv4file
-rw-r–r–. 1 test testgroup 0 Dec 4 2014 nfsv4file

nfs4-test

The file’s permissions are 644, so only the owner will have access to write to the file and groups/others will have read only access. If I want to eliminate “others” from access, I could set that to 640. We can see the user is test@domain.win2k8.netapp.com. But we can’t see what that user’s UID/GID is. The client I am using is connected to LDAP for UID/GID via SSSD (which you can learn more about in TR-4073).

# getent passwd test
test:*:10001:513:test:/home/test:/bin/sh

If I attempt to “spoof” that user but do not know the UID/GID, I won’t have much luck. If I use the same client but cut off the LDAP connection, I can no longer resolve that user:

# service sssd stop
Stopping sssd: [ OK ]
# rm -f /var/lib/sss/db/*
# getent passwd test
#

If the user does not map into my ID domain, I will see nobody:nobody as the owner:group:

# ls -la | grep nfsv4file
-rw-r–r–. 1 nobody nobody 0 Dec 4 14:52 nfsv4file

If I create a user named “test” with a random UID/GID and attempt access to the mount, I’ll still get access denied when trying to write the file. So I can’t really “spoof”  the user unless I have information about that user’s UID/GID or guess correctly.

# getent passwd test
test:x:12345:502::/home/test1:/bin/bash

[root@centos64 /]# su test
bash-4.1$ cd /mnt
bash-4.1$ ls -la | grep nfsv4file
-rw-r–r–. 1 test nobody 0 Dec 4 14:52 nfsv4file
bash-4.1$ touch nfsv4file
touch: cannot touch `nfsv4file’: Permission denied

If I add another layer of abstraction in the form of NFSv4 ACLs, then the mode bits won’t matter, either. And I’ll get to decide which users out of “everyone else” gets access rather than being limited to a single owner and single group.

Adding Kerberos to the mix

If I want to harden my security even further, I can add Kerberos authentication to the NFS configuration. If I use Kerberos, I can set my export policy rules to allow only krb5 (or krb5i) authentication. No more AUTH_SYS allowed! I can use AES-256 bit encryption in clustered Data ONTAP 8.3 and later for the strongest available enctype, which helps combat brute force attacks. I will still be able to see the user string over the wire, but to get to the mount at all, I’d need a valid Kerberos ticket, which would require a valid user principal, password, configured client, DNS entries, etc. In other words, a lot more effort than most people are willing to go through.

Kerberos will add some overhead to performance, as the packet sizes will be bigger and the overall processing will take longer to chew on the encryption bits. But if it’s security you are looking for, NFSv4 + Kerberos is the way to go.

Advertisements

2 thoughts on “TECH::Feeling insecure about NFS?

  1. Pingback: Amazon announces Elastic File System for File Services - Datacenter Dude

  2. Pingback: TECH::How to set up Kerberos on vSphere 6.0 servers for datastores on NFS | Why Is The Internet Broken?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s