PHOTO:: December Monthly Assignment – Christmas Kitsch

This month’s photo assignment was kitsch-y Christmas/holiday decorations. Luckily, I am a kitsch fan, so I have plenty of ridiculous things to choose from. Here are a few…

_DSC7894sm _DSC7885sm _DSC7847sm _DSC7840sm _DSC7835sm _DSC7905sm

Advertisements

TECH::Data LIF best practices for NAS in cDOT 8.3

network_cabling1

NOTE: Some of the documents linked in this blog are in the process of being updated for 8.3, so they may contain older information about clustered Data ONTAP 8.2.x. Check back on occasion to these links to get the most up to date content.

Clustered Data ONTAP 8.3 allows storage administrators to provide the following benefits:

  • Seamless scale-out storage
  • Multiprotocol Unified Access (NFS, CIFS and SAN)
  • Non-disruptive operations

This is done by way of secure multi-tenant architecture with Storage Virtual Machines. This blog will cover logical interface (LIF) considerations for clustered Data ONTAP 8.3 as they pertain to NAS storage operations. These considerations will eventually be added to TR-4067. For a complete list of networking best practices, see TR-4182: Ethernet Storage Best Practices for cDOT Configurations.

Storage Virtual Machines (SVMs)

SVMs are logical storage containers that own storage resources such as flexible volumes, logical interfaces (LIFs), exports, CIFS shares, etc. Think of them as a storage “blade center” in your cluster. These SVMs share physical hardware resources in the cluster with one another, such as network ports/VLANs, aggregates with physical disk, CPU, RAM, switches, etc. As a result, load for SVMs can be balanced across a cluster for maximum performance and efficiency, or to leverage SaaS functionality, among other benefits.

Cluster considerations

A cluster can be comprised of several HA pairs of nodes (4 HA pairs/8 nodes with SAN, 12 HA pairs/24 nodes with NAS). Each node in the cluster has its own copy of a replicated database with the cluster and SVM configuration information. Additionally, each node has its own set of user space applications that handle cluster operations and node-specific caches, not to mention its own set of RAM, CPU, disks, etc. So while a cluster operates as a single entity, it does have the underlying concept of individualized components. As a result, it makes sense to take the physical hardware in a cluster under consideration when implementing and designing.

Data LIF considerations

Data LIFs can live on any physical port in a cluster that is added to a valid broadcast domain. These data LIFs are configured with SVM-aware routing mechanisms that allow for the correct pathing of ethernet traffic in a SVM, regardless of where a valid data LIF lives in the cluster. Prior to 8.3, SVMs routed at a node level, so traffic could only travel via the node that owned a data LIF. In cDOT 8.3, traffic will route from the data LIF even if it is a non-local path.

However, despite this enhancement in cDOT 8.3, it is still worth considering the original best practice recommendation for data LIFs participating in NAS operations…

One data LIF per node, per SVM

With the introduction of IP Spaces in clustered Data ONTAP, this recommendation is more of a reality, as storage administrators no longer have to use unique IP addresses in the cluster for SVMs. With IP Spaces, IP addresses can be duplicated in the cluster on a per-SVM basis to allow for true secure multi-tenancy architecture. For more information on IP Spaces, see TR-4182.

Thus, if using a 24 node cluster, using 24 data LIFs per SVM would be ideal for the following reasons:

  • Ability to leverage data locality features – Features such as NFS referrals, CIFS auto location and pNFS ensure data locality for NAS traffic regardless of where the volumes live in a cluster. These help balance load better, but also make use of local caches and fastpath mechanisms for NAS traffic. For more information, see TR-4067.
  • Ability to reduce cluster network traffic – While cluster network traffic is generally not an issue (you’re more likely to peg the CPU or disk before you saturate a 10gb network), it is better to limit the amount of traffic on a cluster network as much as possible.
  • Ability to ensure data locality in the event of a volume move – If you move a volume to another node, you can ensure you still have a local path to the data if every node has a data LIF for the SVM.
  • Ability to spread the load out across nodes and leverage all the available hardware (CPU, RAM, etc) – If you load up all your NAS traffic on one node via one data LIF, you aren’t realizing the value of the other nodes in the cluster. Spreading network traffic ensures all available physical entities are being utilized. Why pay for hardware you aren’t using?
  • Ability to balance network connections across multiple cluster nodes – Clusters are single entities, as are SVMs. But they do have underlying hardware that have their own maximums, such as number of connections, etc. For info on hardware maximums in cDOT, see the configuration information for your version of ONTAP.
  • Ability to reduce impact of storage failover/givebacks (SFO). – Fewer clients impacted when SFO events happen, whether they are planned or unplanned.
  • Ability to leverage features such as on-box DNS – On-box DNS allows data LIFs to act as DNS servers and honor forwarded zone requests. Once a zone request is received, the cluster will determine the ideal node to service that request based on that node’s CPU and throughput, providing intelligent DNS load balancing (as opposed to round robin DNS, which is a serial process). For more information regarding on-box DNS (and how to configure it), see TR-4182 and TR-4073.

Keep in mind that the above are merely recommendations and not requirements unless using data locality features such as pNFS.

Any questions? Comments? Suggestions for blog topics? Feel free to comment or contact me on Twitter.

TECH::Feeling insecure about NFS?

There’s an old joke about NFS…

not-for-security

And, in many cases, it’s true. NFS versions prior to NFSv4.x did not do much for security. This is true for any client or server.

Some of the issues include:

  • Multiple well-known ports for NFSv3 and prior (NLM, portmapper, mountd, NFS, NSM, rquota)
  • Plain text UID/GID and host information over the wire
  • Kerberos not as useful in NFSv3 (for instance, you cannot Kerberize NLM)

In this blog, I will illustrate just how easy it is to gain access to an exported filesystem with just a little bit of info gathered from a packet capture. I’ll be focusing on clustered Data ONTAP, since that’s what the cool kids are using these days, but these principals can be applied to any NFS implementation, as they adhere to RFC standards.

How NFS access is obtained

NFS access is controlled by two main gateways.

  • Export policies/rules
  • UNIX permissions

Export policies/rules

Export policies/rules are similar to CIFS shares. These simply present the data object to a client for network access. Export policies and rules are covered in great detail in TR-4067: NFS Best Practices and Implementation Guide. These control access based on the client’s IP or hostname rather than by a user, so all one would need to gain access to an export would be an IP address. If I gained access to a network, I could easily change my IP address and hostname to what I see in a trace to “become” that host.

Below is a sample trace where I get exactly that information:

mount-client-info

So now I know the client hostname, the client IP, the NFS server IP and the exported path. Yikes!

In the reply packet, I can see the allowed access, so I know exactly what AUTH flavors I have available to me. Once I see AUTH_UNIX (aka AUTH_SYS), I know I’m in business.

mount-client-info-reply

With AUTH_SYS, I know I don’t need a Kerberos ticket to get in to the volume. All I need is the client that has root access to do some real damage.

In this case, my export policy rule is set to be pretty wide open. In this case, I am not practicing good security. I just want ease of access. Below is the sample of the policy rule (on clustered Data ONTAP):

cluster::*> export-policy rule show -vserver SVM -policyname wideopen -instance
(vserver export-policy rule show)

Vserver: SVM
Policy Name: wideopen
Rule Index: 1
Access Protocol: any
Client Match Hostname, IP Address, Netgroup, or Domain: 0.0.0.0/0
RO Access Rule: any
RW Access Rule: any
User ID To Which Anonymous Users Are Mapped: 0
Superuser Security Types: any
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: unrestricted
Vserver Change Ownership Mode: unrestricted

So with the above policy and NFSv3, all anyone would need is to know *a* client IP address in my network. It’s much better to limit the client match to a hostname, IP or netgroup to ensure that someone attempting access would at least need to make the effort to sniff packets. I may also want to specify an AUTH type, preferably Kerberos (we’ll touch on that later). I’d also be much better off by not setting my “superuser” rule to “any” which allows carte blanche root access to anyone logging in as root on a client. Oh, and definitely don’t want “anon” to be 0. This one is much better:

cluster::*> export-policy rule show -vserver SVM -policyname map-anon -instance
(vserver export-policy rule show)

Vserver: SVM
Policy Name: map-anon
Rule Index: 1
Access Protocol: any
Client Match Hostname, IP Address, Netgroup, or Domain: 10.228.225.140
RO Access Rule: any
RW Access Rule: any
User ID To Which Anonymous Users Are Mapped: test
Superuser Security Types: none
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: unrestricted

Vserver: SVM
Policy Name: map-anon
Rule Index: 2
Access Protocol: any
Client Match Hostname, IP Address, Netgroup, or Domain: 10.228.225.128
RO Access Rule: none
RW Access Rule: none
User ID To Which Anonymous Users Are Mapped: test
Superuser Security Types: none
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: unrestricted
2 entries were displayed.

Now I have allowed access to the volume to only two IP addresses. I’ve also set these clients to squash root to the anon user, which is not UID 0. So, much more secure, but we can do better.

UNIX Permissions

The next level of security for NFS after exports would be UNIX permissions. These are the file/directory level access rights for users and groups.

In NFSv3, these are very limited. You essentially get owner, group and everyone else. That means you can’t assign multiple groups to a file or folder unless you nest the groups. And multiple owners?

Fuggedaboutit.

photo_james_gandolfini-500x373

Additionally, with NFSv3, we face the same issue we faced with hostnames over the wire – I can see what the file permissions are and what UID/GID I would need to gain access.

In the trace excerpt below, I can see in a LOOKUP reply, my file has 755 permissions and the owner/group is 0:0. So all I need to do is become root. Good thing I locked down root in my export policy above!

nfs-access

But I still have “5” (or read) access to everyone else. So all I need to do is become any old user to gain access to read this file. Maybe this file has the salaries of everyone at my company. Maybe it has the secret recipe for Coca Cola. Or maybe you have Excel files with passwords

……………………………………..________
………………………………,.-‘”……………….“~.,
………………………..,.-“……………………………..”-.,
…………………….,/………………………………………..”:,
…………………,?………………………………………………,
………………./…………………………………………………..,}
……………../………………………………………………,:`^`..}
……………/……………………………………………,:”………/
…………..?…..__…………………………………..:`………../
…………./__.(…..”~-,_…………………………,:`………./
………../(_….”~,_……..”~,_………………..,:`…….._/
……….{.._$;_……”=,_…….”-,_…….,.-~-,},.~”;/….}
………..((…..*~_…….”=-._……”;,,./`…./”…………../
…,,,___.`~,……”~.,………………..`…..}…………../
…………(….`=-,,…….`……………………(……;_,,-”
…………/.`~,……`-………………………………./
………….`~.*-,……………………………….|,./…..,__
,,_……….}.>-._……………………………..|…………..`=~-,
…..`=~-,__……`,……………………………
……………….`=~-,,.,………………………….
…………………………..`:,,………………………`…………..__
……………………………….`=-,……………….,%`>–==“
…………………………………._……….._,-%…….`
……………………………..,

Who knows, but it sure would be easy to find out at this point!

Security Lockdown: NFSv4 with Kerberos

There are two main ways to lock down NFS outside of export policies and mode bit permissions: NFSv4 and Kerberos. NFSv4 provides the following security benefits over NFSv3:

  • Single port used for NFS (all other protocol pieces live on this same port); less ports to open on firewalls
  • Identity domain strings to reduce UID/GID visibility and to enforce domain ID mapping for proper authentication
  • Better integration with Kerberos/AUTH_GSS
  • NFSV4 ACLs – more granularity for permissions than mode bits

NOTE: NFSv4.x is covered in more detail in TRs 3580, 4067 and 4073.

In the following packet capture, when I look for information about the UID/GID, I see the following:

nfs4-access

In the above, the permissions are still 755. And I see that the owner is root, but I also see that root has to belong to the NFSv4 ID domain of domain.win2k8.netapp.com. I can still spoof this user (especially when it’s root) by modifying the client to leverage NFSv4 ID domains, etc. But for users that I do not have the UID/GID for, it’s much harder.

Consider this scenario… I have a file called “nfsv4file.” It is owned by test@domain.win2k8.netapp.com.

# ls -la | grep nfsv4file
-rw-r–r–. 1 test testgroup 0 Dec 4 2014 nfsv4file

nfs4-test

The file’s permissions are 644, so only the owner will have access to write to the file and groups/others will have read only access. If I want to eliminate “others” from access, I could set that to 640. We can see the user is test@domain.win2k8.netapp.com. But we can’t see what that user’s UID/GID is. The client I am using is connected to LDAP for UID/GID via SSSD (which you can learn more about in TR-4073).

# getent passwd test
test:*:10001:513:test:/home/test:/bin/sh

If I attempt to “spoof” that user but do not know the UID/GID, I won’t have much luck. If I use the same client but cut off the LDAP connection, I can no longer resolve that user:

# service sssd stop
Stopping sssd: [ OK ]
# rm -f /var/lib/sss/db/*
# getent passwd test
#

If the user does not map into my ID domain, I will see nobody:nobody as the owner:group:

# ls -la | grep nfsv4file
-rw-r–r–. 1 nobody nobody 0 Dec 4 14:52 nfsv4file

If I create a user named “test” with a random UID/GID and attempt access to the mount, I’ll still get access denied when trying to write the file. So I can’t really “spoof”  the user unless I have information about that user’s UID/GID or guess correctly.

# getent passwd test
test:x:12345:502::/home/test1:/bin/bash

[root@centos64 /]# su test
bash-4.1$ cd /mnt
bash-4.1$ ls -la | grep nfsv4file
-rw-r–r–. 1 test nobody 0 Dec 4 14:52 nfsv4file
bash-4.1$ touch nfsv4file
touch: cannot touch `nfsv4file’: Permission denied

If I add another layer of abstraction in the form of NFSv4 ACLs, then the mode bits won’t matter, either. And I’ll get to decide which users out of “everyone else” gets access rather than being limited to a single owner and single group.

Adding Kerberos to the mix

If I want to harden my security even further, I can add Kerberos authentication to the NFS configuration. If I use Kerberos, I can set my export policy rules to allow only krb5 (or krb5i) authentication. No more AUTH_SYS allowed! I can use AES-256 bit encryption in clustered Data ONTAP 8.3 and later for the strongest available enctype, which helps combat brute force attacks. I will still be able to see the user string over the wire, but to get to the mount at all, I’d need a valid Kerberos ticket, which would require a valid user principal, password, configured client, DNS entries, etc. In other words, a lot more effort than most people are willing to go through.

Kerberos will add some overhead to performance, as the packet sizes will be bigger and the overall processing will take longer to chew on the encryption bits. But if it’s security you are looking for, NFSv4 + Kerberos is the way to go.

COMICS::Random Hulk Musings

hulk

Now for something near and dear to my heart… The Incredible Hulk. It may even be a bit of an unhealthy obsession. See this picture? That’s me behind the mask…

hulk-insight

If you don’t know who the Hulk is (shame on you), it is the alter-ego of Bruce Banner, spawned from a bath of gamma radiation from a bomb test gone wrong…

panel_hulk001b

On a deeper level, it’s about control. It’s about embracing the true you. Banner, in several storylines, either loses control completely or learns to harness the Hulk and be at one with him.

Psychologically, the Hulk is Banner’s id amplified and personified, with Banner’s ego and super-ego wrestling for control. So while the Hulk is generally perceived as a simple beast (think: Hulk smash), he’s actually very complex and interesting.

In fact, you could argue that the Hulk is not actually chaos, but chaos theory.

the-credible-hulk

I digress… the real impetus for this blog post is something that occurred to me that I have not seen mentioned before or referenced in any story lines. If you’ve thought of the same or have seen it in print, please let me know so I can go read!

Put aside for a moment the fact that a living being could survive the sheer science behind gamma radiation exposure and accept that the Hulk could actually exist.

Where is the fallout? (pun intended)

The Hulk is green. He started out grey and has switched back and forth several times through the years. But his best known form is green. Why is that important? Well, green denotes that he was hit by gamma, which has been colorized as green. (Let’s ignore that green is generally associated with envy or greed or even peace…)

Green here means he’s infected with gamma radiation. Perhaps even extreme radiation poisoning. Wouldn’t all of that green-ness mean he’s also emitting at least *some* gamma radiation? There’s evidence in the comics that he is radioactive:

radioactivehulk

One explanation is that his blood is radioactive, which is how we got She-Hulk. But wouldn’t that radiation pass through Hulk’s skin, and even Banner’s in human form?

And wouldn’t that radiation become amplified as he gets angrier? The Red Hulk gets hotter as he gets angrier, and it even takes a toll on his physical well-being. Why wouldn’t the same happen to the Hulk?

With that in mind, wouldn’t that radiation affect people in his life like Betty Ross? As we saw in the Watchmen, Ozymandias plotted to make people believe that Dr. Manhattan had given people in his life cancer. And people believed it because it was credible!

manhattan

We also see various radioactive-based characters (covered very nicely in this fantastic blog post: America’s Radioactive Superheroes) throughout DC and Marvel comics that wear special suits to either contain or protect others from their radiation. Does the Hulk’s skin act as a radiation shield? Or is this simply a plot hole in the character?

Perhaps it’s a yet unexplored storyline… maybe even yet to be unlocked powers.