Why Is the Internet Broken: Greatest Hits

When I started this site back in October of 2014, it was mainly to drive traffic to my NetApp Insight sessions -and it worked.

(By the way… stay tuned for a blog on this year’s new Insight sessions by yours truly. Now with more lab!)

As I continued writing, my goal was to keep creating content – don’t be the guy who just shows up during conference season.

blogfieldofdreams

So far, so good.

But since I create so much content, it gets hard to find for new visitors to this site, The WordPress archives/table of contents is lacking. So, what I’ve done is create my own table of contents of the top 5 most visited posts.

Top 5 Blogs (by number of visits)

TECH::Using NFS with Docker – Where does it fit in?

SMB1 Vulnerabilities: How do they affect NetApp’s Data ONTAP?

TECH::Become a clustered Data ONTAP CLI Ninja

ONTAP 9.1 is now generally available (GA)!

NetApp FlexGroup: An evolution of NAS

DataCenterDude

I also used to write for datacenterdude.com on occasion.

To read those, go to this link:

My DataCenterDude stuff

How else do I find stuff?

You can also search on the site or click through the archives, if you choose. Or, subscribe to the RSS feed. If you have questions or want to see something changed or added to the site, follow me on Twitter @NFSDudeAbides or comment on one of the posts here!

You can also email me at whyistheinternetbroken@gmail.com.

Advertisements

Behind the Scenes: Episode 157 – Performance Analysis Using OnCommand Unified Manager

Welcome to the Episode 157, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we welcome Mr. Performance himself, Tony Gaddis (gaddis@netapp.com) to give us a tutorial on easily finding performance issues using OnCommand Unified Manager, as well as some common “rules of thumb” when it comes to how much latency and node utilization is too much.

Also, check out Tony’s NetApp Insight 2018 session in Las Vegas and Barcelona:

1181-1 – ONTAP Storage Performance Design Considerations for Emerging Technologies

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 156 – SnapCenter 4.1

Welcome to the Episode 156, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, the SnapCenter Wonder Twins  – TMEs Steven Cortez (@mscproductions) and Jeanine Walter (@j9Walter) join us to give the lowdown on what’s new in the latest SnapCenter release. 

You can find SnapCenter 4.1 here: 

https://mysupport.netapp.com/NOW/download/software/snapcenter/4.1/  

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 155 – Trident 18.07 and… Goodbye, Sully?

Welcome to the Episode 155, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

podcast-dust.png

This week on the podcast, we make Sully do work on his last day at NetApp, as we discuss what’s new in NetApp Trident 18.07. We also have some special guests – Jonathan Rippy (@jkrippy) and Garrett Mueller (@innergy) – stop by to wish Sully well as he heads off to RedHat. We’ll miss you, Sully!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 154 – Max DATA

Welcome to the Episode 154, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we talk about the latest and greatest NetApp product, MAXData! Join the MAXData evangelists as they give the rundown on what it is, where you’d use it and how screaming fast it is. Featured in this episode:

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 153 – VVols and VMware Best Practices in ONTAP

Welcome to the Episode 153, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, ONTAP Virtualization TME Karl Konnerth (@konnerth) joins us to give us the latest in VMware best practices for ONTAP, as well as a primer on VVols and what the latest functionality is in vSphere 7.2! 

For more information, read Karl’s recent blog about VVols & ONTAP or check out the updated technical report

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 152 – VMworld 2018 Preview + NetApp HCI Update

Welcome to the Episode 152, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, VMworld 2018 is fast approaching! Join me and a group of NetApp’s finest virtualization experts to discuss what’s in store for NetApp at VMworld 2018, as well as what we can expect for NetApp HCI (https://www.netapp.com/us/products/converged-systems/hyper-converged-infrastructure.aspx).

On this episode:

  • Doug Chamberlain (@dacmba), Senior Product Marketing Manager
  • Keith Norbie (@keithnorbie), Senior Business Development Manager
  • Gabriel Chapman (@Bacon_Is_King), Senior Manager, NetApp HCI
  • Karl Konnerth (@konnerth), ONTAP Virtualization TME

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Securing NFS mounts in a Docker container

docker_security-e1530093759599

Setting up Kerberized NFS on a client can be a bit challenging, especially if you’re trying to do it across multiple hosts. So, I decided I wanted to take on the challenge of creating an easy to deploy Docker container, using NetApp’s Trident plugin to make life even easier.

Why do I want Kerberos?

dockersecurity-476x192

With Kerberos on NFS mounts, you can encrypt traffic for authentication (krb5), integrity checking (krb5i) and for end-to end packet encryption (krb5p). I covered the benefits of using krb5p in a previous blog. I also covered how to use NFS with the new FlexGroup driver in Docker + NFS + FlexGroup volumes = Magic!

I also cover krb5p in Encrypt your NFS packets end to end with krb5p and ONTAP 9.2!

But this blog covers FlexVols only, since FlexGroup volumes can’t use NFSv4.x – yet.

Why do I want NFSv4.x?

NFSv3 is a great protocol, but it has some disadvantages when it comes to locking and security. For starters, v3 is stateless. NFSv4.x is stateful and manages locks much better, since it’s done on a lease basis and is integrated in the protocol itself, while v3 has ancillary services that manage locks.

Those ancillary services (like NLM, mountd, portmap) are also what makes NFSv3 less secure than NFSv4.x. More services = more ports to open on a firewall (your network guy hates you, btw). Additionally, standard in-flight encryption for NAS protocols, such as Kerberos, don’t encrypt the ancillary services – it only encrypts the NFS packets. NFSv4.x also has additional layers of security via NFSv4.x ACLs, as well as ID domain name mapping to ensure the client and server agree on who is who when accessing NFS mounts.

The main downside of NFSv4.x is performance. It currently lags behind NFSv3 for a variety of reasons, but mostly because it has to do more in each packet than NFSv3 had to, and being a stateful protocol can be costly to performance. When you lump in encryption, it adds more overhead. Things are getting better, however, and soon, there won’t be any excuse for not using NFSv4.x as your standard NFS version.

What you need before you start

In this example, I’m going to configure Kerberos, NFSv4.1 and LDAP on a single container. This allows me to have all the moving parts I’d need to get it working out of the gate. I’m using CentOS7.x/RHEL7.x as the Docker host and container base, as well as Microsoft Active Directory 2012R2 for LDAP UNIX identities and Kerberos KDC functionality. You can use Linux-based LDAP and KDCs, but that’s outside the scope of what this blog is about.

Before you get started, you need the following.

  • Active Directory configured to use LDAP for UNIX identity mapping
  • A server/VM running the latest CentOS/RHEL version (our Docker host)
  • A NetApp ONTAP cluster with a SVM running NFS on it

Configuring the ONTAP SVM

Before you can get started with NFS Kerberos on the client, you’ll need to configure NFS Kerberos in ONTAP. Doing this essentially comes down to the following steps:

  • Create a Kerberos realm
  • Configure DNS
  • AES encryption types allowed on the NFS server
  • Create a Kerberos interface (this creates a machine object in AD that has your SPN for NFS server ticket functionality and adds the keytab to the cluster/SVM)
  • Create a local UNIX user named “nfs” (this maps to the nfs/service account when Kerberos mounts are attempted)
  • Create a generic name mapping rule for all machine accounts (when you join a container to the Kerberos realm, it creates a new machine account with the format of [imagehexname]$@REALM.COM. Having a generic name mapping rule will eliminate headaches trying to manage that)
  • Create an export policy and rule that allows Kerberos authentication/v4.x for NFS
  • (optional, but recommended) LDAP server configuration that matches the client (this makes life much easier with NFSv4.x)
  • Configure the NFSv4 ID domain option and enable NFSv4.0/4.1

These steps are all covered in pretty good detail in TR-4073 (the unabridged version) and TR-4616 (the more streamlined version), so I won’t cover them here. Instead, I’ll show you how my cluster is configured.

Kerberos realm

::*> kerberos realm show -vserver DEMO -instance
(vserver nfs kerberos realm show)

Vserver: DEMO
Kerberos Realm: NTAP.LOCAL
KDC Vendor: Microsoft
KDC IP Address: 10.x.x.x
KDC Port: 88
Clock Skew: 5
Active Directory Server Name: oneway.ntap.local
Active Directory Server IP Address: 10.x.x.x
Comment: -
Admin Server IP Address: 10.x.x.x
Admin Server Port: 749
Password Server IP Address: 10.x.x.x
Password Server Port: 464
Permitted Encryption Types: aes-128, aes-256

DNS

::*> dns show -vserver DEMO -instance

Vserver: DEMO
Domains: NTAP.LOCAL
Name Servers: 10.x.x.x
Timeout (secs): 5
Maximum Attempts: 1
Is TLD Query Enabled?: true
Require Source and Reply IPs to Match: true
Require Packet Queries to Match: true

AES encryption types allowed on the NFS server

::*> nfs server show -vserver DEMO -fields permitted-enc-types
vserver permitted-enc-types
------- -------------------
DEMO aes-128,aes-256

Kerberos interface

::*> kerberos interface show -vserver DEMO -instance
(vserver nfs kerberos interface show)

Vserver: DEMO
Logical Interface: data
IP Address: 10.x.x.x
Kerberos Enabled: enabled
Service Principal Name: nfs/demo.ntap.local@NTAP.LOCAL
Permitted Encryption Types: aes-128, aes-256
Machine Account Name: -

Vserver: DEMO
Logical Interface: data2
IP Address: 10.x.x.x
Kerberos Enabled: enabled
Service Principal Name: nfs/demo.ntap.local@NTAP.LOCAL
Permitted Encryption Types: aes-128, aes-256
Machine Account Name: -
2 entries were displayed.

UNIX user named NFS

::*> unix-user show -vserver DEMO -user nfs -instance

Vserver: DEMO
User Name: nfs
User ID: 500
Primary Group ID: 500
User's Full Name:

Generic name mapping rule for Kerberos SPNs

::*> vserver name-mapping show -vserver DEMO -direction krb-unix -instance

Vserver: DEMO
Direction: krb-unix
Position: 1
Pattern: (.+)\$@NTAP.LOCAL
Replacement: root
IP Address with Subnet Mask: -
Hostname: -

Export policy rule that allows Kerberos/NFSv4.x

::*> export-policy rule show -vserver DEMO -policyname kerberos -instance

Vserver: DEMO
Policy Name: kerberos
Rule Index: 1
Access Protocol: nfs4
List of Client Match Hostnames, IP Addresses, Netgroups, or Domains: 0/0
RO Access Rule: krb5, krb5i, krb5p
RW Access Rule: krb5, krb5i, krb5p
User ID To Which Anonymous Users Are Mapped: 65534
Superuser Security Types: any
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: use_export_policy
Policy ID: 42949672971

LDAP client config (optional, but recommended if you plan to use NFSv4.x)

::*> ldap client show -client-config DEMO -instance

Vserver: DEMO
Client Configuration Name: DEMO
LDAP Server List: -
(DEPRECATED)-LDAP Server List: -
Active Directory Domain: NTAP.LOCAL
Preferred Active Directory Servers: -
Bind Using the Vserver's CIFS Credentials: true
Schema Template: MS-AD-BIS
LDAP Server Port: 389
Query Timeout (sec): 3
Minimum Bind Authentication Level: sasl
Bind DN (User): mtuser
Base DN: DC=NTAP,DC=local
Base Search Scope: subtree
User DN: -
User Search Scope: subtree
Group DN: -
Group Search Scope: subtree
Netgroup DN: -
Netgroup Search Scope: subtree
Vserver Owns Configuration: true
Use start-tls Over LDAP Connections: false
Enable Netgroup-By-Host Lookup: false
Netgroup-By-Host DN: -
Netgroup-By-Host Scope: subtree
Client Session Security: none
LDAP Referral Chasing: false
Group Membership Filter: -

To test LDAP functionality on the cluster, use the following command in advanced privilege to look up a user. If you get a UID/GID, you’re good to go.

::*> getxxbyyy getpwbyname -node ontap9-tme-8040-01 -vserver DEMO -username prof1
(vserver services name-service getxxbyyy getpwbyname)
pw_name: prof1
pw_passwd:
pw_uid: 1100
pw_gid: 1101
pw_gecos:
pw_dir:
pw_shell:

Configure/enable NFSv4.x

::*> nfs server show -vserver DEMO -fields v4-id-domain,v4.1,v4.0
vserver v4.0 v4-id-domain v4.1
------- ------- ------------ -------
DEMO enabled NTAP.LOCAL enabled

Once the cluster SVM is set up, there shouldn’t be much else, if anything, that needs to be done for Kerberos on the cluster. However, in AD, you’ll want to allow only AES for the NFS server machine account with this simple PowerShell command:

PS C:\> Set-ADComputer NFS-KRB-NAME$ -KerberosEncryptionType AES256,AES128

Configuring the Docker host

To get NFSv4.x to work properly in a container, you’ll need to make a decision about your Docker host. What I found in my testing is that containers running NFSv4.x want to use the Docker host’s ID mappings/users when doing NFSv4.x functions, rather than the container’s. So while the container may be able to pull users from LDAP and write files as those users, you will see NFSv4.x owners and groups that the Docker *host* cannot resolve appear as “nobody.”

sh-4.2$ ls -la
total 8
drwxrwxrwx. 2 root root 4096 Aug 14 21:08 .
drwxr-xr-x. 18 root root 4096 Aug 15 15:06 ..
-rw-r--r--. 1 nobody nobody 0 Aug 13 21:38 newfile

So, if you want NFSv4.x to resolve names properly (and I suspect you do), then you need to do one of the following on the Docker host:

a) Add users and groups locally to the passwd/group files

b) Configure SSSD to query LDAP

Naturally, I like things to be consistent, so I chose option b.

Since we already have an LDAP server in AD, we can just install/configure sssd to use that. Here’s what you’d do…

Install necessary packages

I like using realm and sssd. It’s fun. It’s easy. This is what you need to do that.

yum -y install realmd sssd oddjob oddjob-mkhomedir adcli samba-common krb5-workstation ntp

Configure DNS (/etc/resolv.conf)

This needs to point to the name servers in your AD domain.

Create a generic machine account in AD for LDAP authentication

This will be how our LDAP clients bind. You can use it for more than one client if you like. This can be done via PowerShell.

PS C:\> import-module activedirectory
PS C:\> New-ADComputer -Name [computername] -SAMAccountName [computername] -DNSHostName
[computername.dns.domain.com] -OtherAttributes @{'userAccountControl'= 2097152;'msDSSupportedEncryptionTypes'=27}

Create a keytab file to copy to the Docker host to use for LDAP binds

This is done on the AD domain controller in a CLI window. Use ktpass and this syntax:

ktpass -princ primary/instance@REALM -mapuser [DOMAIN]\machine$ -crypto AES256-SHA1 +rndpass -ptype KRB5_NT_PRINCIPAL +Answer -out [file:\location]

Copy the keytab file to your Docker host

WinSCP is a good tool to do this. It should live as /etc/krb5.keytab

Configure /etc/krb5.conf

Here’s an example (changes in bold):

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 30d
renew_lifetime = 30d
forwardable = true
rdns = false
# default_realm = EXAMPLE.COM
default_ccache_name = KEYRING:persistent:%{uid}

default_realm = NTAP.LOCAL
[realms]
# EXAMPLE.COM = {
# kdc = kerberos.example.com
# admin_server = kerberos.example.com
# }

NTAP.LOCAL = {
}

[domain_realm]
# .example.com = EXAMPLE.COM
# example.com = EXAMPLE.COM
ntap.local = NTAP.LOCAL
.ntap.local = NTAP.LOCAL

Configure the sssd.conf file to point to LDAP

This is what mine looks like. Note the LDAP URI and SASL authid. I also set “use_fully_qualified_names” to false.

[domain/default]
cache_credentials = False
case_sensitive = False
enumerate = True

[sssd]
config_file_version = 2
services = nss, pam, autofs
domains = NTAP.local
debug_level = 7
[nss]
filter_users = root,ldap,named,avahi,haldaemon,dbus,radiusd,news,nscd
filter_groups = root
[pam]
[domain/DOMAIN]
auth_provider = krb5
chpass_provider = krb5
id_provider = ldap
ldap_search_base = dc=ntap,dc=local
ldap_schema = rfc2307bis
ldap_sasl_mech = GSSAPI
ldap_user_object_class = user
ldap_group_object_class = group
ldap_user_home_directory = unixHomeDirectory
ldap_user_principal = userPrincipalName
ldap_account_expire_policy = ad
ldap_force_upper_case_realm = true
ldap_user_search_base = cn=Users,dc=ntap,dc=local
ldap_group_search_base = cn=Users,dc=ntap,dc=local
ldap_sasl_authid = root/krb-container.ntap.local@NTAP.LOCAL
krb5_server = ntap.local
krb5_realm = NTAP.LOCAL
krb5_kpasswd = ntap.local
use_fully_qualified_names = false

Enable authconfig and start SSSD

authconfig --enablesssd --enablesssdauth --updateall

systemctl start sssd

Test LDAP functionality/name lookup

You can use “getent” or “id” to look names up.

# id prof1
uid=1100(prof1) gid=1101(ProfGroup) groups=1101(ProfGroup),1203(group3),1202(group2),1201(group1),1220(sharedgroup)

Configure /etc/idmapd.conf with the NFSv4.x domain

Just this single line needs to be added. Needs to match what’s on the ONTAP SVM.

Domain = [NTAP.LOCAL]

Creating your container

I used the centos/http:latest as the base image, and am running with systemd. I also am copying a few config files to the container to ensure it functions properly and then running a script afterwards.

Here’s the dockerfile I used to create a container that could do NFSv4.x, Kerberos and LDAP. You can also find it on GitHub here:

https://github.com/whyistheinternetbroken/centos-kerberos-nfsv4-sssd

FROM centos/httpd:latest
ENV container docker

# Copy the dbus.service file from systemd to location with Dockerfile
ADD dbus.service /usr/lib/systemd/system/dbus.service

VOLUME ["/sys/fs/cgroup"]
VOLUME ["/run"]

CMD ["/usr/lib/systemd/systemd"]

RUN yum -y install centos-release-scl-rh && \
yum -y install --setopt=tsflags=nodocs mod_ssl
RUN yum -y update; yum clean all
RUN yum -y install --setopt=tsflags=nodocs sssd sssd-dbus adcli krb5-workstation ntp realmd oddjob oddjob-mkhomedir samba-common samba-common-tools nfs-utils; yum clean all

## Systemd cleanup base image
RUN (cd /lib/systemd/system/sysinit.target.wants && for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -vf $i; done) & \
rm -vf /lib/systemd/system/multi-user.target.wants/* && \
rm -vf /etc/systemd/system/*.wants/* && \
rm -vf /lib/systemd/system/local-fs.target.wants/* && \
rm -vf /lib/systemd/system/sockets.target.wants/*udev* && \
rm -vf /lib/systemd/system/sockets.target.wants/*initctl* && \
rm -vf /lib/systemd/system/basic.target.wants/* && \
rm -vf /lib/systemd/system/anaconda.target.wants/*

# Copy the local SSSD conf file
RUN mkdir -p /etc/sssd
COPY sssd.conf /etc/sssd/sssd.conf

# Copy the local krb files
COPY krb5.keytab /etc/krb5.keytab
COPY krb5.conf /etc/krb5.conf

# Copy the NFSv4 IDmap file
COPY idmapd.conf /etc/idmapd.conf

#Copy the DNS config
COPY resolv.conf /etc/resolv.conf

# Copy rc.local
COPY rc.local /etc/rc.d/rc.local

# start services
ADD configure-nfs.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/configure-nfs.sh
RUN chmod +x /etc/rc.d/rc.local

You’ll notice that I have several COPY commands in the file. Those are config files you’d need to modify to reflect your own environment. You’ll want to store these in the same folder as your dockerfile. I’ll break down each file here.

dbus.service

This file is the same file as the one on the Docker host. It allows the container to run with systemd. Simply copy it from /usr/lib/systemd/system/dbus.service into your dockerfile folder location.

sssd.conf

This is our LDAP file and specifies how LDAP does its queries. While NFSv4.x will use the Docker host’s users for NFSv4.x mapping, the container will still need to know who a user is to allow us to su, kinit, etc. For this, you can essentially use the same config file you used for your Docker host.

krb5.keytab and krb5.conf

The krb5.keytab file is used to authenticate/bind to LDAP only in this case. So, use the same keytab file you created earlier. Same for the krb5.conf file, unless your containers are going to leverage a different KDC/domain than the Docker host. In that case, it gets a little more complicated. Just copy the Docker host’s krb5.keytab and krb5.conf files from /etc.

idmapd.conf

Again, same file as the Docker host. This defines our idmap domain for NFSv4.x.

resolv.conf

DNS information; should match what’s on the Docker host.

rc.local

This file is useful for running our configuration script. We need the script to run because the container won’t let you start services before it’s running. When you try, you get this error (or something similar):

Failed to get D-Bus connection: No connection to service manager.

This is the line I added to my rc.local:

/usr/local/bin/configure-nfs.sh

That leads us to the script…

configure-nfs.sh

This script starts services. It also joins the container to the Kerberos realm. While I’m using AD KDCs, you can also use realm join to join Linux-based KDCs. Maybe one day I’ll set one up and write up a guide, but for now, read the Linux KDC docs. 🙂

For the realm join, I’m passing the password with the command. It’s in plaintext, so I’d recommend not using a domain admin here. Realm join uses administrator by default, but you have a way to specify a different user with the -U option. So, you can either create a user that *only* has access to create/delete objects in a specific OU or leave the password portion out and have users enter the password when the container starts.

I’d also highly recommend creating a new OU in AD to house all your container machine objects. Otherwise, you’ll see your default OU get flooded with these:

krb-containers

So, configure an OU or CN in AD and then point realm join to use that.

docker-ou.png

Here’s my shell script:

#!/bin/sh
systemctl start dbus
systemctl start rpcgssd
systemctl start rpcidmapd
systemctl restart sssd
echo PASSWORD| realm join -U username --computer-ou OU=Docker NTAP.LOCAL

Realm join caveat

In my config, I’ve done something “clever.” When you join a realm on a Linux client, it will also configure SSSD to pull UNIX IDs from AD. It doesn’t use the uid field by default. Instead, it creates a UID based on the AD SID. Thus, user student1 might look like this from LDAP (as expected):

# id student1
uid=1301(student1) gid=1201(group1) groups=1201(group1),1220(sharedgroup),1203(group3)

But would look like this from SSSD’s algorithm:

# id student1@NTAP.LOCAL
uid=1587401108(student1@NTAP.local) gid=1587400513(domainusers@NTAP.local) groups=1587400513(domainusers@NTAP.local),1587401107(group3@NTAP.local),1587401105(group1@NTAP.local),1587401122(sharedgroup@NTAP.local)

ONTAP doesn’t really know how to query UIDs in the way SSSD does, so we’d need SSSD to be able to look up our UNIX users, but also be able to query AD users that may not have UNIX attributes populated. To control that, I set my sssd.conf file to do the following:

  • When a username is specified without a FQDN, SSSD looks it up in normal LDAP
  • When a username is specified with a FQDN, SSSD uses the algorithm

I controlled this with the SSSD option use_fully_qualified_names. I set it to false for my UNIX users. When realm join is run, it appends to the sssd.conf file and uses the default value of use_fully_qualified_names, which is “true.”

Here’s what realmd adds to the file:

[domain/NTAP.local]
ad_domain = NTAP.local
krb5_realm = NTAP.LOCAL
realmd_tags = manages-system joined-with-samba
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = ad

Build your container!

That’s pretty much it. Once you have your Docker host and ONTAP cluster configured, Kerberizing NFS in containers is a breeze. Simply build your Docker container using the dockerfile:

docker build -f /dockerfiles/dockerfile.kerb -t parisi/centos-krb-client .

And then run it in privileged mode. The following also shows us specifying a volume that has been created using NetApp Trident. (see below for my Trident config.json file)

docker run --rm -it --privileged -d -v kerberos:/kerberos parisi/centos-krb-client

And then you can exec the container and start using Kerberos!

# docker exec -ti 330e10f7db1d bash

# su student1
sh-4.2$ klist
klist: Credentials cache keyring 'persistent:1301:1301' not found
sh-4.2$ kinit
Password for student1@NTAP.LOCAL:
sh-4.2$ klist -e
Ticket cache: KEYRING:persistent:1301:1301
Default principal: student1@NTAP.LOCAL

Valid starting Expires Service principal
08/16/18 14:52:58 08/17/18 00:52:58 krbtgt/NTAP.LOCAL@NTAP.LOCAL
renew until 08/23/18 14:52:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.2$ cd /kerberos
sh-4.2$ klist -e
Ticket cache: KEYRING:persistent:1301:1301
Default principal: student1@NTAP.LOCAL

Valid starting Expires Service principal
08/16/18 14:53:09 08/17/18 00:52:58 nfs/demo.ntap.local@NTAP.LOCAL
renew until 08/23/18 14:52:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
08/16/18 14:52:58 08/17/18 00:52:58 krbtgt/NTAP.LOCAL@NTAP.LOCAL
renew until 08/23/18 14:52:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.2$ ls -la
total 8
drwxrwxrwx. 2 root root 4096 Aug 15 15:46 .
drwxr-xr-x. 18 root root 4096 Aug 15 19:02 ..
-rw-r--r--. 1 prof1 ProfGroup 0 Aug 15 15:41 newfile
-rw-r--r--. 1 student1 group1 0 Aug 14 21:08 newfile2
-rw-r--r--. 1 root daemon 0 Aug 15 15:41 newfile3
-rw-r--r--. 1 student1 group1 0 Aug 15 15:46 newfile4
-rw-r--r--. 1 prof1 ProfGroup 0 Aug 13 20:57 prof1file
-rw-r--r--. 1 student1 group1 0 Aug 13 20:58 student1
-rw-r--r--. 1 student2 group2 0 Aug 13 21:12 student2

You can also push the docker image up to your repository and pull it down on any Docker host you like, provided that Docker host is configured as we mentioned.

BONUS ROUND: Trident config.json

tumblr_lndumwicey1qdh05go1_400

Did you know you could control mount options and export policy rules with Trident?

Get Trident here!

Just use the config.json file to do that. In my file, every volume mounts as NFSv4.1 with Kerberos security and a Kerberos export policy.

{
"version": 1,
"storageDriverName": "ontap-nas",
"managementLIF": "10.x.x.x",
"dataLIF": "10.x.x.x",
"svm": "DEMO",
"username": "admin",
"password": "PASSWORD",
"aggregate": "aggr1_node1",
"exportPolicy": "kerberos",
"nfsMountOptions": "-o vers=4.1,sec=krb5",
"defaults": {
"exportPolicy": "kerberos"
}
}

Happy Kerberizing!

Behind the Scenes: Episode 151 – Honeycomb.io, Serverless Concepts and Dev/Ops

Welcome to the Episode 151, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Docker + NFS + FlexGroup volumes = Magic!

tapete-as-creation-the-magic-unicorns-8-470937_l

A couple of years ago, I wrote up a blog on using NFS with Docker as I was tooling around with containers, in an attempt to wrap my head around them. Then, I never really touched them again and that blog got a bit… stale.

Why stale?

Well, in that blog, I had to create a bunch of kludgy hacks to get NFS to work with Docker, and honestly, it likely wasn’t even the best way to do it, given my lack of overall Docker knowledge. More recently, I wrote up a way to Kerberize NFS mounts in Docker containers that is a little better effort.

Luckily, realizing that I’m not the only one who wants to use Docker but may not know all the ins and outs, NetApp developers created a NetApp plugin to use with Docker that will do all the volume creation, removal, etc for you. Then, you can leverage the Docker volume options to mount via NFS. That plugin is named “Trident.”

mattel-dc-multiverse-super-friends-aquaman-review-trident-2

Trident + NFS

Trident is an open source storage provisioner and orchestrator for the NetApp portfolio.

You can read more about it here:

https://netapp.io/2016/12/23/introducing-trident-dynamic-persistent-volume-provisioner-kubernetes/

You can also read about how we use it for AI/ML here:

https://www.theregister.co.uk/2018/08/03/netapp_a800_pure_airi_flashblade/

When you’re using the Trident plugin, you can create Docker-ready NFS exported volumes in ONTAP to provide storage to all of your containers just by specifying the -v option during your “docker run” commands.

For example, here’s a NFS exported volume created using the Trident plugin:

# docker volume create -d netapp --name=foo_justin
foo_justin
# docker volume ls
DRIVER VOLUME NAME
netapp:latest foo_justin

Here’s what shows up on the ONTAP system:

::*> vol show -vserver DEMO -volume netappdvp_foo_justin -fields policy
vserver volume               policy
------- -------------------- -------
DEMO    netappdvp_foo_justin default

Then, I can just start up the container using that volume:

# docker run --rm -it -v foo_justin:/foo alpine ash
/ # mount | grep justin
10.x.x.x:/netappdvp_foo_justin on /foo type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.193.67.237,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.x.x.x)

Having a centralized NFS storage volume for your containers to rely on has a vast number of use cases, providing access for reading and writing to the same location across a network on a high-performing storage system with all sorts of data protection capabilities to ensure high availability and resiliency.

Customization of Volumes

With the Trident plugin, you have the ability to modify the config files to change attributes from the defaults, such as custom names, size, export policies and others. See the full list here:

http://netapp-trident.readthedocs.io/en/latest/docker/install/ndvp_ontap_config.html

Trident + NFS + FlexGroup Volumes

Starting in Trident 18.07, a new Trident NAS driver was added that supports creation of FlexGroup volumes with Docker.

To change the plugin, change the /etc/netappdvp/config.json file to use the FlexGroup driver.

{
"version": 1,
"storageDriverName": "ontap-nas-flexgroup",
"managementLIF": "10.x.x.x",
"dataLIF": "10.x.x.x.",
"svm": "DEMO",
"username": "admin",
"password": "********",
"aggregate": "aggr1_node1",
}

Then, create your FlexGroup volume. That simple!

A word of advice, though. The FlexGroup driver defaults to 1GB and creates 8 member volumes across your aggregates, which creates 128MB member volumes. That’s problematic for a couple reasons:

  • FlexGroup volumes should have members that are no less than 100GB in size (as per TR-4571) – small members will affect performance due to member volumes doing more remote allocation than normal
  • Files that get written to the FlexGroup will fill up 128MB pretty fast, causing the FlexGroup to appear to be out of space.

You can fix this either by setting the config.json file to use larger sizes, or specifying the size up front in the Docker volume command. I’d recommend using the config file and overriding the defaults.

To set this in the config file, just specify “size” as a variable (full list of options can be found here: https://netapp-trident.readthedocs.io/en/latest/kubernetes/operations/tasks/backends/ontap.html:

{
    "version": 1,
    "storageDriverName": "ontap-nas-flexgroup",
    "managementLIF": "10.0.0.1",
    "dataLIF": "10.0.0.2",
    "svm": "svm_nfs",
    "username": "vsadmin",
    "password": "secret",
    "defaults": {
      "size": "800G",
      "spaceReserve": "volume",
      "exportPolicy": "myk8scluster"
    }}

Since the volumes default to thin provisioned, you shouldn’t worry too much about storage space, unless you think your clients will fill up 800GB. If that’s the case, you can apply quotas to the volumes if needed to limit how much space can be used. (For FlexGroups, quota enforcement will be available in an upcoming release; FlexVols can currently use quota enforcement)

# docker volume create -d netapp --name=foo_justin_fg -o size=1t
foo_justin_fg

And this is what the volume looks like in ONTAP:

::*> vol show -vserver DEMO -volume netappdvp_foo_justin* -fields policy,is-flexgroup,aggr-list,size,space-guarantee 
vserver volume                  aggr-list               size policy  space-guarantee is-flexgroup
------- ----------------------- ----------------------- ---- ------- --------------- ------------
DEMO netappdvp_foo_justin_fg    aggr1_node1,aggr1_node2 1TB  default none            true

Since the FlexGroup is 1TB in size, the member volumes will be 128GB, which fulfills the 100GB minimum. Future releases will enforce this without you having to worry about it.

::*> vol show -vserver DEMO -volume netappdvp_foo_justin_fg_* -fields aggr-list,size -sort-by aggr-list
vserver volume                        aggr-list   size
------- ----------------------------- ----------- -----
DEMO    netappdvp_foo_justin_fg__0001 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0003 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0005 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0007 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0002 aggr1_node2 128GB
DEMO    netappdvp_foo_justin_fg__0004 aggr1_node2 128GB
DEMO    netappdvp_foo_justin_fg__0006 aggr1_node2 128GB
DEMO    netappdvp_foo_justin_fg__0008 aggr1_node2 128GB
8 entries were displayed.

Practical uses for FlexGroups with containers

It’s cool that we *can* provision FlexGroup volumes with Trident for use with containers, but does that mean we should?

Well, consider this…

In an ONTAP cluster that uses FlexVol volumes for NFS storage presented to containers, I am going to be bound to a single node’s resources, as per the design of a FlexVol. This means that even though I bought a 4 node cluster, I can only use 1 node’s RAM, CPU, network, capacity, etc. If I have a use case where thousands of containers spin up at any given moment and attach themselves to a NFS volume, then I might see some performance bottlenecks due to the increased load. In most cases, that’s fine – but if you could get more out of your storage, wouldn’t you want to do that?

docker-flexvol

You could add layers of automation into the mix to add more FlexVols to the solution, but then you have new mount points/folders. And what if those containers all need to access the same data?

docker-flexvol2

With a FlexGroup volume that gets presented to those same Docker instances, the containers now can leverage all nodes in the cluster, use a single namespace and simplify the overall automation structure.

docker-flexgroup.png

The benefits become even more evident when those containers are constantly writing new files to the NFS mount, such as in an Artificial Intelligence/Machine Learning use case. FlexGroups were designed to handle massive amounts of file creations and can provide 2-6x the performance over a FlexVol in use cases where we’re constantly creating new files.

Stay tuned for some more information on how FlexGroups and Trident can bring even more capability to the table to AI/ML workloads. In the meantime, you can learn more about NetApp solutions for AI/ML here:

https://www.netapp.com/us/solutions/applications/ai-deep-learning.aspx

Behind the Scenes: Episode 150 – Tech Career Advancement Advice

Welcome to the Episode 150, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we change it up a bit and bring in VMware Global Executive Content Strategist, Tech LevelUp founder, Sunnyvale VMUG leader, Tech Village podcaster and all around community all-star, Yadin Porter de Leon (@porterdeleon) to talk about Tech Careers and how to advance. We also talk about combating imposter syndrome and overcoming your fears.

You can find the Tech Village podcast here:

https://soundcloud.com/techvillage

Tech LevelUp here:

https://twitter.com/Tech_LevelUp

VMUG here:

http://vmug.com

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here: