Behind the Scenes: Episode 159 – NetApp at AnsibleFest

Welcome to the Episode 159, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, NetApp TME David Blackwell and Sr. Product and Program Manage Lawrence Bunka join us to discuss what NetApp will be up to at AnsibleFest.

Also, be sure to check out this Ansible session at NetApp Insight 2018! 

  • 1128-3 – Day 0/1 Configuration Management Using Ansible for NetApp ONTAP  

You can also sign up to join the NetApp Pub Slack channel at netapp.io! 

You can find David Blackwell’s NetApp Ansible container here:

https://hub.docker.com/r/schmots1/netapp-ansible/  

Finding the Podcast

You can find this week’s episode here:

https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/506757057%3Fsecret_token%3Ds-BYliS&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 155 – Trident 18.07 and… Goodbye, Sully?

Welcome to the Episode 155, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

podcast-dust.png

This week on the podcast, we make Sully do work on his last day at NetApp, as we discuss what’s new in NetApp Trident 18.07. We also have some special guests – Jonathan Rippy (@jkrippy) and Garrett Mueller (@innergy) – stop by to wish Sully well as he heads off to RedHat. We’ll miss you, Sully!

Finding the Podcast

You can find this week’s episode here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Securing NFS mounts in a Docker container

docker_security-e1530093759599

Setting up Kerberized NFS on a client can be a bit challenging, especially if you’re trying to do it across multiple hosts. So, I decided I wanted to take on the challenge of creating an easy to deploy Docker container, using NetApp’s Trident plugin to make life even easier.

Why do I want Kerberos?

dockersecurity-476x192

With Kerberos on NFS mounts, you can encrypt traffic for authentication (krb5), integrity checking (krb5i) and for end-to end packet encryption (krb5p). I covered the benefits of using krb5p in a previous blog. I also covered how to use NFS with the new FlexGroup driver in Docker + NFS + FlexGroup volumes = Magic!

I also cover krb5p in Encrypt your NFS packets end to end with krb5p and ONTAP 9.2!

But this blog covers FlexVols only, since FlexGroup volumes can’t use NFSv4.x – yet.

Why do I want NFSv4.x?

NFSv3 is a great protocol, but it has some disadvantages when it comes to locking and security. For starters, v3 is stateless. NFSv4.x is stateful and manages locks much better, since it’s done on a lease basis and is integrated in the protocol itself, while v3 has ancillary services that manage locks.

Those ancillary services (like NLM, mountd, portmap) are also what makes NFSv3 less secure than NFSv4.x. More services = more ports to open on a firewall (your network guy hates you, btw). Additionally, standard in-flight encryption for NAS protocols, such as Kerberos, don’t encrypt the ancillary services – it only encrypts the NFS packets. NFSv4.x also has additional layers of security via NFSv4.x ACLs, as well as ID domain name mapping to ensure the client and server agree on who is who when accessing NFS mounts.

The main downside of NFSv4.x is performance. It currently lags behind NFSv3 for a variety of reasons, but mostly because it has to do more in each packet than NFSv3 had to, and being a stateful protocol can be costly to performance. When you lump in encryption, it adds more overhead. Things are getting better, however, and soon, there won’t be any excuse for not using NFSv4.x as your standard NFS version.

What you need before you start

In this example, I’m going to configure Kerberos, NFSv4.1 and LDAP on a single container. This allows me to have all the moving parts I’d need to get it working out of the gate. I’m using CentOS7.x/RHEL7.x as the Docker host and container base, as well as Microsoft Active Directory 2012R2 for LDAP UNIX identities and Kerberos KDC functionality. You can use Linux-based LDAP and KDCs, but that’s outside the scope of what this blog is about.

Before you get started, you need the following.

  • Active Directory configured to use LDAP for UNIX identity mapping
  • A server/VM running the latest CentOS/RHEL version (our Docker host)
  • A NetApp ONTAP cluster with a SVM running NFS on it

Configuring the ONTAP SVM

Before you can get started with NFS Kerberos on the client, you’ll need to configure NFS Kerberos in ONTAP. Doing this essentially comes down to the following steps:

  • Create a Kerberos realm
  • Configure DNS
  • AES encryption types allowed on the NFS server
  • Create a Kerberos interface (this creates a machine object in AD that has your SPN for NFS server ticket functionality and adds the keytab to the cluster/SVM)
  • Create a local UNIX user named “nfs” (this maps to the nfs/service account when Kerberos mounts are attempted)
  • Create a generic name mapping rule for all machine accounts (when you join a container to the Kerberos realm, it creates a new machine account with the format of [imagehexname]$@REALM.COM. Having a generic name mapping rule will eliminate headaches trying to manage that)
  • Create an export policy and rule that allows Kerberos authentication/v4.x for NFS
  • (optional, but recommended) LDAP server configuration that matches the client (this makes life much easier with NFSv4.x)
  • Configure the NFSv4 ID domain option and enable NFSv4.0/4.1

These steps are all covered in pretty good detail in TR-4073 (the unabridged version) and TR-4616 (the more streamlined version), so I won’t cover them here. Instead, I’ll show you how my cluster is configured.

Kerberos realm

::*> kerberos realm show -vserver DEMO -instance
(vserver nfs kerberos realm show)

Vserver: DEMO
Kerberos Realm: NTAP.LOCAL
KDC Vendor: Microsoft
KDC IP Address: 10.x.x.x
KDC Port: 88
Clock Skew: 5
Active Directory Server Name: oneway.ntap.local
Active Directory Server IP Address: 10.x.x.x
Comment: -
Admin Server IP Address: 10.x.x.x
Admin Server Port: 749
Password Server IP Address: 10.x.x.x
Password Server Port: 464
Permitted Encryption Types: aes-128, aes-256

DNS

::*> dns show -vserver DEMO -instance

Vserver: DEMO
Domains: NTAP.LOCAL
Name Servers: 10.x.x.x
Timeout (secs): 5
Maximum Attempts: 1
Is TLD Query Enabled?: true
Require Source and Reply IPs to Match: true
Require Packet Queries to Match: true

AES encryption types allowed on the NFS server

::*> nfs server show -vserver DEMO -fields permitted-enc-types
vserver permitted-enc-types
------- -------------------
DEMO aes-128,aes-256

Kerberos interface

::*> kerberos interface show -vserver DEMO -instance
(vserver nfs kerberos interface show)

Vserver: DEMO
Logical Interface: data
IP Address: 10.x.x.x
Kerberos Enabled: enabled
Service Principal Name: nfs/demo.ntap.local@NTAP.LOCAL
Permitted Encryption Types: aes-128, aes-256
Machine Account Name: -

Vserver: DEMO
Logical Interface: data2
IP Address: 10.x.x.x
Kerberos Enabled: enabled
Service Principal Name: nfs/demo.ntap.local@NTAP.LOCAL
Permitted Encryption Types: aes-128, aes-256
Machine Account Name: -
2 entries were displayed.

UNIX user named NFS

::*> unix-user show -vserver DEMO -user nfs -instance

Vserver: DEMO
User Name: nfs
User ID: 500
Primary Group ID: 500
User's Full Name:

Generic name mapping rule for Kerberos SPNs

::*> vserver name-mapping show -vserver DEMO -direction krb-unix -instance

Vserver: DEMO
Direction: krb-unix
Position: 1
Pattern: (.+)\$@NTAP.LOCAL
Replacement: root
IP Address with Subnet Mask: -
Hostname: -

Export policy rule that allows Kerberos/NFSv4.x

::*> export-policy rule show -vserver DEMO -policyname kerberos -instance

Vserver: DEMO
Policy Name: kerberos
Rule Index: 1
Access Protocol: nfs4
List of Client Match Hostnames, IP Addresses, Netgroups, or Domains: 0/0
RO Access Rule: krb5, krb5i, krb5p
RW Access Rule: krb5, krb5i, krb5p
User ID To Which Anonymous Users Are Mapped: 65534
Superuser Security Types: any
Honor SetUID Bits in SETATTR: true
Allow Creation of Devices: true
NTFS Unix Security Options: fail
Vserver NTFS Unix Security Options: use_export_policy
Change Ownership Mode: restricted
Vserver Change Ownership Mode: use_export_policy
Policy ID: 42949672971

LDAP client config (optional, but recommended if you plan to use NFSv4.x)

::*> ldap client show -client-config DEMO -instance

Vserver: DEMO
Client Configuration Name: DEMO
LDAP Server List: -
(DEPRECATED)-LDAP Server List: -
Active Directory Domain: NTAP.LOCAL
Preferred Active Directory Servers: -
Bind Using the Vserver's CIFS Credentials: true
Schema Template: MS-AD-BIS
LDAP Server Port: 389
Query Timeout (sec): 3
Minimum Bind Authentication Level: sasl
Bind DN (User): mtuser
Base DN: DC=NTAP,DC=local
Base Search Scope: subtree
User DN: -
User Search Scope: subtree
Group DN: -
Group Search Scope: subtree
Netgroup DN: -
Netgroup Search Scope: subtree
Vserver Owns Configuration: true
Use start-tls Over LDAP Connections: false
Enable Netgroup-By-Host Lookup: false
Netgroup-By-Host DN: -
Netgroup-By-Host Scope: subtree
Client Session Security: none
LDAP Referral Chasing: false
Group Membership Filter: -

To test LDAP functionality on the cluster, use the following command in advanced privilege to look up a user. If you get a UID/GID, you’re good to go.

::*> getxxbyyy getpwbyname -node ontap9-tme-8040-01 -vserver DEMO -username prof1
(vserver services name-service getxxbyyy getpwbyname)
pw_name: prof1
pw_passwd:
pw_uid: 1100
pw_gid: 1101
pw_gecos:
pw_dir:
pw_shell:

Configure/enable NFSv4.x

::*> nfs server show -vserver DEMO -fields v4-id-domain,v4.1,v4.0
vserver v4.0 v4-id-domain v4.1
------- ------- ------------ -------
DEMO enabled NTAP.LOCAL enabled

Once the cluster SVM is set up, there shouldn’t be much else, if anything, that needs to be done for Kerberos on the cluster. However, in AD, you’ll want to allow only AES for the NFS server machine account with this simple PowerShell command:

PS C:\> Set-ADComputer NFS-KRB-NAME$ -KerberosEncryptionType AES256,AES128

Configuring the Docker host

To get NFSv4.x to work properly in a container, you’ll need to make a decision about your Docker host. What I found in my testing is that containers running NFSv4.x want to use the Docker host’s ID mappings/users when doing NFSv4.x functions, rather than the container’s. So while the container may be able to pull users from LDAP and write files as those users, you will see NFSv4.x owners and groups that the Docker *host* cannot resolve appear as “nobody.”

sh-4.2$ ls -la
total 8
drwxrwxrwx. 2 root root 4096 Aug 14 21:08 .
drwxr-xr-x. 18 root root 4096 Aug 15 15:06 ..
-rw-r--r--. 1 nobody nobody 0 Aug 13 21:38 newfile

So, if you want NFSv4.x to resolve names properly (and I suspect you do), then you need to do one of the following on the Docker host:

a) Add users and groups locally to the passwd/group files

b) Configure SSSD to query LDAP

Naturally, I like things to be consistent, so I chose option b.

Since we already have an LDAP server in AD, we can just install/configure sssd to use that. Here’s what you’d do…

Install necessary packages

I like using realm and sssd. It’s fun. It’s easy. This is what you need to do that.

yum -y install realmd sssd oddjob oddjob-mkhomedir adcli samba-common krb5-workstation ntp

Configure DNS (/etc/resolv.conf)

This needs to point to the name servers in your AD domain.

Create a generic machine account in AD for LDAP authentication

This will be how our LDAP clients bind. You can use it for more than one client if you like. This can be done via PowerShell.

PS C:\> import-module activedirectory
PS C:\> New-ADComputer -Name [computername] -SAMAccountName [computername] -DNSHostName
[computername.dns.domain.com] -OtherAttributes @{'userAccountControl'= 2097152;'msDSSupportedEncryptionTypes'=27}

Create a keytab file to copy to the Docker host to use for LDAP binds

This is done on the AD domain controller in a CLI window. Use ktpass and this syntax:

ktpass -princ primary/instance@REALM -mapuser [DOMAIN]\machine$ -crypto AES256-SHA1 +rndpass -ptype KRB5_NT_PRINCIPAL +Answer -out [file:\location]

Copy the keytab file to your Docker host

WinSCP is a good tool to do this. It should live as /etc/krb5.keytab

Configure /etc/krb5.conf

Here’s an example (changes in bold):

# Configuration snippets may be placed in this directory as well
includedir /etc/krb5.conf.d/

[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
dns_lookup_realm = false
ticket_lifetime = 30d
renew_lifetime = 30d
forwardable = true
rdns = false
# default_realm = EXAMPLE.COM
default_ccache_name = KEYRING:persistent:%{uid}

default_realm = NTAP.LOCAL
[realms]
# EXAMPLE.COM = {
# kdc = kerberos.example.com
# admin_server = kerberos.example.com
# }

NTAP.LOCAL = {
}

[domain_realm]
# .example.com = EXAMPLE.COM
# example.com = EXAMPLE.COM
ntap.local = NTAP.LOCAL
.ntap.local = NTAP.LOCAL

Configure the sssd.conf file to point to LDAP

This is what mine looks like. Note the LDAP URI and SASL authid. I also set “use_fully_qualified_names” to false.

[domain/default]
cache_credentials = False
case_sensitive = False
enumerate = True

[sssd]
config_file_version = 2
services = nss, pam, autofs
domains = NTAP.local
debug_level = 7
[nss]
filter_users = root,ldap,named,avahi,haldaemon,dbus,radiusd,news,nscd
filter_groups = root
[pam]
[domain/DOMAIN]
auth_provider = krb5
chpass_provider = krb5
id_provider = ldap
ldap_search_base = dc=ntap,dc=local
ldap_schema = rfc2307bis
ldap_sasl_mech = GSSAPI
ldap_user_object_class = user
ldap_group_object_class = group
ldap_user_home_directory = unixHomeDirectory
ldap_user_principal = userPrincipalName
ldap_account_expire_policy = ad
ldap_force_upper_case_realm = true
ldap_user_search_base = cn=Users,dc=ntap,dc=local
ldap_group_search_base = cn=Users,dc=ntap,dc=local
ldap_sasl_authid = root/krb-container.ntap.local@NTAP.LOCAL
krb5_server = ntap.local
krb5_realm = NTAP.LOCAL
krb5_kpasswd = ntap.local
use_fully_qualified_names = false

Enable authconfig and start SSSD

authconfig --enablesssd --enablesssdauth --updateall

systemctl start sssd

Test LDAP functionality/name lookup

You can use “getent” or “id” to look names up.

# id prof1
uid=1100(prof1) gid=1101(ProfGroup) groups=1101(ProfGroup),1203(group3),1202(group2),1201(group1),1220(sharedgroup)

Configure /etc/idmapd.conf with the NFSv4.x domain

Just this single line needs to be added. Needs to match what’s on the ONTAP SVM.

Domain = [NTAP.LOCAL]

Creating your container

I used the centos/http:latest as the base image, and am running with systemd. I also am copying a few config files to the container to ensure it functions properly and then running a script afterwards.

Here’s the dockerfile I used to create a container that could do NFSv4.x, Kerberos and LDAP. You can also find it on GitHub here:

https://github.com/whyistheinternetbroken/centos-kerberos-nfsv4-sssd

FROM centos/httpd:latest
ENV container docker

# Copy the dbus.service file from systemd to location with Dockerfile
ADD dbus.service /usr/lib/systemd/system/dbus.service

VOLUME ["/sys/fs/cgroup"]
VOLUME ["/run"]

CMD ["/usr/lib/systemd/systemd"]

RUN yum -y install centos-release-scl-rh && \
yum -y install --setopt=tsflags=nodocs mod_ssl
RUN yum -y update; yum clean all
RUN yum -y install --setopt=tsflags=nodocs sssd sssd-dbus adcli krb5-workstation ntp realmd oddjob oddjob-mkhomedir samba-common samba-common-tools nfs-utils; yum clean all

## Systemd cleanup base image
RUN (cd /lib/systemd/system/sysinit.target.wants && for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -vf $i; done) & \
rm -vf /lib/systemd/system/multi-user.target.wants/* && \
rm -vf /etc/systemd/system/*.wants/* && \
rm -vf /lib/systemd/system/local-fs.target.wants/* && \
rm -vf /lib/systemd/system/sockets.target.wants/*udev* && \
rm -vf /lib/systemd/system/sockets.target.wants/*initctl* && \
rm -vf /lib/systemd/system/basic.target.wants/* && \
rm -vf /lib/systemd/system/anaconda.target.wants/*

# Copy the local SSSD conf file
RUN mkdir -p /etc/sssd
COPY sssd.conf /etc/sssd/sssd.conf

# Copy the local krb files
COPY krb5.keytab /etc/krb5.keytab
COPY krb5.conf /etc/krb5.conf

# Copy the NFSv4 IDmap file
COPY idmapd.conf /etc/idmapd.conf

#Copy the DNS config
COPY resolv.conf /etc/resolv.conf

# Copy rc.local
COPY rc.local /etc/rc.d/rc.local

# start services
ADD configure-nfs.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/configure-nfs.sh
RUN chmod +x /etc/rc.d/rc.local

You’ll notice that I have several COPY commands in the file. Those are config files you’d need to modify to reflect your own environment. You’ll want to store these in the same folder as your dockerfile. I’ll break down each file here.

dbus.service

This file is the same file as the one on the Docker host. It allows the container to run with systemd. Simply copy it from /usr/lib/systemd/system/dbus.service into your dockerfile folder location.

sssd.conf

This is our LDAP file and specifies how LDAP does its queries. While NFSv4.x will use the Docker host’s users for NFSv4.x mapping, the container will still need to know who a user is to allow us to su, kinit, etc. For this, you can essentially use the same config file you used for your Docker host.

krb5.keytab and krb5.conf

The krb5.keytab file is used to authenticate/bind to LDAP only in this case. So, use the same keytab file you created earlier. Same for the krb5.conf file, unless your containers are going to leverage a different KDC/domain than the Docker host. In that case, it gets a little more complicated. Just copy the Docker host’s krb5.keytab and krb5.conf files from /etc.

idmapd.conf

Again, same file as the Docker host. This defines our idmap domain for NFSv4.x.

resolv.conf

DNS information; should match what’s on the Docker host.

rc.local

This file is useful for running our configuration script. We need the script to run because the container won’t let you start services before it’s running. When you try, you get this error (or something similar):

Failed to get D-Bus connection: No connection to service manager.

This is the line I added to my rc.local:

/usr/local/bin/configure-nfs.sh

That leads us to the script…

configure-nfs.sh

This script starts services. It also joins the container to the Kerberos realm. While I’m using AD KDCs, you can also use realm join to join Linux-based KDCs. Maybe one day I’ll set one up and write up a guide, but for now, read the Linux KDC docs. 🙂

For the realm join, I’m passing the password with the command. It’s in plaintext, so I’d recommend not using a domain admin here. Realm join uses administrator by default, but you have a way to specify a different user with the -U option. So, you can either create a user that *only* has access to create/delete objects in a specific OU or leave the password portion out and have users enter the password when the container starts.

I’d also highly recommend creating a new OU in AD to house all your container machine objects. Otherwise, you’ll see your default OU get flooded with these:

krb-containers

So, configure an OU or CN in AD and then point realm join to use that.

docker-ou.png

Here’s my shell script:

#!/bin/sh
systemctl start dbus
systemctl start rpcgssd
systemctl start rpcidmapd
systemctl restart sssd
echo PASSWORD| realm join -U username --computer-ou OU=Docker NTAP.LOCAL

Realm join caveat

In my config, I’ve done something “clever.” When you join a realm on a Linux client, it will also configure SSSD to pull UNIX IDs from AD. It doesn’t use the uid field by default. Instead, it creates a UID based on the AD SID. Thus, user student1 might look like this from LDAP (as expected):

# id student1
uid=1301(student1) gid=1201(group1) groups=1201(group1),1220(sharedgroup),1203(group3)

But would look like this from SSSD’s algorithm:

# id student1@NTAP.LOCAL
uid=1587401108(student1@NTAP.local) gid=1587400513(domainusers@NTAP.local) groups=1587400513(domainusers@NTAP.local),1587401107(group3@NTAP.local),1587401105(group1@NTAP.local),1587401122(sharedgroup@NTAP.local)

ONTAP doesn’t really know how to query UIDs in the way SSSD does, so we’d need SSSD to be able to look up our UNIX users, but also be able to query AD users that may not have UNIX attributes populated. To control that, I set my sssd.conf file to do the following:

  • When a username is specified without a FQDN, SSSD looks it up in normal LDAP
  • When a username is specified with a FQDN, SSSD uses the algorithm

I controlled this with the SSSD option use_fully_qualified_names. I set it to false for my UNIX users. When realm join is run, it appends to the sssd.conf file and uses the default value of use_fully_qualified_names, which is “true.”

Here’s what realmd adds to the file:

[domain/NTAP.local]
ad_domain = NTAP.local
krb5_realm = NTAP.LOCAL
realmd_tags = manages-system joined-with-samba
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = True
use_fully_qualified_names = True
fallback_homedir = /home/%u@%d
access_provider = ad

Build your container!

That’s pretty much it. Once you have your Docker host and ONTAP cluster configured, Kerberizing NFS in containers is a breeze. Simply build your Docker container using the dockerfile:

docker build -f /dockerfiles/dockerfile.kerb -t parisi/centos-krb-client .

And then run it in privileged mode. The following also shows us specifying a volume that has been created using NetApp Trident. (see below for my Trident config.json file)

docker run --rm -it --privileged -d -v kerberos:/kerberos parisi/centos-krb-client

And then you can exec the container and start using Kerberos!

# docker exec -ti 330e10f7db1d bash

# su student1
sh-4.2$ klist
klist: Credentials cache keyring 'persistent:1301:1301' not found
sh-4.2$ kinit
Password for student1@NTAP.LOCAL:
sh-4.2$ klist -e
Ticket cache: KEYRING:persistent:1301:1301
Default principal: student1@NTAP.LOCAL

Valid starting Expires Service principal
08/16/18 14:52:58 08/17/18 00:52:58 krbtgt/NTAP.LOCAL@NTAP.LOCAL
renew until 08/23/18 14:52:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.2$ cd /kerberos
sh-4.2$ klist -e
Ticket cache: KEYRING:persistent:1301:1301
Default principal: student1@NTAP.LOCAL

Valid starting Expires Service principal
08/16/18 14:53:09 08/17/18 00:52:58 nfs/demo.ntap.local@NTAP.LOCAL
renew until 08/23/18 14:52:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
08/16/18 14:52:58 08/17/18 00:52:58 krbtgt/NTAP.LOCAL@NTAP.LOCAL
renew until 08/23/18 14:52:55, Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
sh-4.2$ ls -la
total 8
drwxrwxrwx. 2 root root 4096 Aug 15 15:46 .
drwxr-xr-x. 18 root root 4096 Aug 15 19:02 ..
-rw-r--r--. 1 prof1 ProfGroup 0 Aug 15 15:41 newfile
-rw-r--r--. 1 student1 group1 0 Aug 14 21:08 newfile2
-rw-r--r--. 1 root daemon 0 Aug 15 15:41 newfile3
-rw-r--r--. 1 student1 group1 0 Aug 15 15:46 newfile4
-rw-r--r--. 1 prof1 ProfGroup 0 Aug 13 20:57 prof1file
-rw-r--r--. 1 student1 group1 0 Aug 13 20:58 student1
-rw-r--r--. 1 student2 group2 0 Aug 13 21:12 student2

You can also push the docker image up to your repository and pull it down on any Docker host you like, provided that Docker host is configured as we mentioned.

BONUS ROUND: Trident config.json

tumblr_lndumwicey1qdh05go1_400

Did you know you could control mount options and export policy rules with Trident?

Get Trident here!

Just use the config.json file to do that. In my file, every volume mounts as NFSv4.1 with Kerberos security and a Kerberos export policy.

{
"version": 1,
"storageDriverName": "ontap-nas",
"managementLIF": "10.x.x.x",
"dataLIF": "10.x.x.x",
"svm": "DEMO",
"username": "admin",
"password": "PASSWORD",
"aggregate": "aggr1_node1",
"exportPolicy": "kerberos",
"nfsMountOptions": "-o vers=4.1,sec=krb5",
"defaults": {
"exportPolicy": "kerberos"
}
}

Happy Kerberizing!

Docker + NFS + FlexGroup volumes = Magic!

tapete-as-creation-the-magic-unicorns-8-470937_l

A couple of years ago, I wrote up a blog on using NFS with Docker as I was tooling around with containers, in an attempt to wrap my head around them. Then, I never really touched them again and that blog got a bit… stale.

Why stale?

Well, in that blog, I had to create a bunch of kludgy hacks to get NFS to work with Docker, and honestly, it likely wasn’t even the best way to do it, given my lack of overall Docker knowledge. More recently, I wrote up a way to Kerberize NFS mounts in Docker containers that is a little better effort.

Luckily, realizing that I’m not the only one who wants to use Docker but may not know all the ins and outs, NetApp developers created a NetApp plugin to use with Docker that will do all the volume creation, removal, etc for you. Then, you can leverage the Docker volume options to mount via NFS. That plugin is named “Trident.”

mattel-dc-multiverse-super-friends-aquaman-review-trident-2

Trident + NFS

Trident is an open source storage provisioner and orchestrator for the NetApp portfolio.

You can read more about it here:

https://netapp.io/2016/12/23/introducing-trident-dynamic-persistent-volume-provisioner-kubernetes/

You can also read about how we use it for AI/ML here:

https://www.theregister.co.uk/2018/08/03/netapp_a800_pure_airi_flashblade/

When you’re using the Trident plugin, you can create Docker-ready NFS exported volumes in ONTAP to provide storage to all of your containers just by specifying the -v option during your “docker run” commands.

For example, here’s a NFS exported volume created using the Trident plugin:

# docker volume create -d netapp --name=foo_justin
foo_justin
# docker volume ls
DRIVER VOLUME NAME
netapp:latest foo_justin

Here’s what shows up on the ONTAP system:

::*> vol show -vserver DEMO -volume netappdvp_foo_justin -fields policy
vserver volume               policy
------- -------------------- -------
DEMO    netappdvp_foo_justin default

Then, I can just start up the container using that volume:

# docker run --rm -it -v foo_justin:/foo alpine ash
/ # mount | grep justin
10.x.x.x:/netappdvp_foo_justin on /foo type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.193.67.237,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.x.x.x)

Having a centralized NFS storage volume for your containers to rely on has a vast number of use cases, providing access for reading and writing to the same location across a network on a high-performing storage system with all sorts of data protection capabilities to ensure high availability and resiliency.

Customization of Volumes

With the Trident plugin, you have the ability to modify the config files to change attributes from the defaults, such as custom names, size, export policies and others. See the full list here:

http://netapp-trident.readthedocs.io/en/latest/docker/install/ndvp_ontap_config.html

Trident + NFS + FlexGroup Volumes

Starting in Trident 18.07, a new Trident NAS driver was added that supports creation of FlexGroup volumes with Docker.

To change the plugin, change the /etc/netappdvp/config.json file to use the FlexGroup driver.

{
"version": 1,
"storageDriverName": "ontap-nas-flexgroup",
"managementLIF": "10.x.x.x",
"dataLIF": "10.x.x.x.",
"svm": "DEMO",
"username": "admin",
"password": "********",
"aggregate": "aggr1_node1",
}

Then, create your FlexGroup volume. That simple!

A word of advice, though. The FlexGroup driver defaults to 1GB and creates 8 member volumes across your aggregates, which creates 128MB member volumes. That’s problematic for a couple reasons:

  • FlexGroup volumes should have members that are no less than 100GB in size (as per TR-4571) – small members will affect performance due to member volumes doing more remote allocation than normal
  • Files that get written to the FlexGroup will fill up 128MB pretty fast, causing the FlexGroup to appear to be out of space.

You can fix this either by setting the config.json file to use larger sizes, or specifying the size up front in the Docker volume command. I’d recommend using the config file and overriding the defaults.

To set this in the config file, just specify “size” as a variable (full list of options can be found here: https://netapp-trident.readthedocs.io/en/latest/kubernetes/operations/tasks/backends/ontap.html:

{
    "version": 1,
    "storageDriverName": "ontap-nas-flexgroup",
    "managementLIF": "10.0.0.1",
    "dataLIF": "10.0.0.2",
    "svm": "svm_nfs",
    "username": "vsadmin",
    "password": "secret",
    "defaults": {
      "size": "800G",
      "spaceReserve": "volume",
      "exportPolicy": "myk8scluster"
    }}

Since the volumes default to thin provisioned, you shouldn’t worry too much about storage space, unless you think your clients will fill up 800GB. If that’s the case, you can apply quotas to the volumes if needed to limit how much space can be used. (For FlexGroups, quota enforcement will be available in an upcoming release; FlexVols can currently use quota enforcement)

# docker volume create -d netapp --name=foo_justin_fg -o size=1t
foo_justin_fg

And this is what the volume looks like in ONTAP:

::*> vol show -vserver DEMO -volume netappdvp_foo_justin* -fields policy,is-flexgroup,aggr-list,size,space-guarantee 
vserver volume                  aggr-list               size policy  space-guarantee is-flexgroup
------- ----------------------- ----------------------- ---- ------- --------------- ------------
DEMO netappdvp_foo_justin_fg    aggr1_node1,aggr1_node2 1TB  default none            true

Since the FlexGroup is 1TB in size, the member volumes will be 128GB, which fulfills the 100GB minimum. Future releases will enforce this without you having to worry about it.

::*> vol show -vserver DEMO -volume netappdvp_foo_justin_fg_* -fields aggr-list,size -sort-by aggr-list
vserver volume                        aggr-list   size
------- ----------------------------- ----------- -----
DEMO    netappdvp_foo_justin_fg__0001 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0003 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0005 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0007 aggr1_node1 128GB
DEMO    netappdvp_foo_justin_fg__0002 aggr1_node2 128GB
DEMO    netappdvp_foo_justin_fg__0004 aggr1_node2 128GB
DEMO    netappdvp_foo_justin_fg__0006 aggr1_node2 128GB
DEMO    netappdvp_foo_justin_fg__0008 aggr1_node2 128GB
8 entries were displayed.

Practical uses for FlexGroups with containers

It’s cool that we *can* provision FlexGroup volumes with Trident for use with containers, but does that mean we should?

Well, consider this…

In an ONTAP cluster that uses FlexVol volumes for NFS storage presented to containers, I am going to be bound to a single node’s resources, as per the design of a FlexVol. This means that even though I bought a 4 node cluster, I can only use 1 node’s RAM, CPU, network, capacity, etc. If I have a use case where thousands of containers spin up at any given moment and attach themselves to a NFS volume, then I might see some performance bottlenecks due to the increased load. In most cases, that’s fine – but if you could get more out of your storage, wouldn’t you want to do that?

docker-flexvol

You could add layers of automation into the mix to add more FlexVols to the solution, but then you have new mount points/folders. And what if those containers all need to access the same data?

docker-flexvol2

With a FlexGroup volume that gets presented to those same Docker instances, the containers now can leverage all nodes in the cluster, use a single namespace and simplify the overall automation structure.

docker-flexgroup.png

The benefits become even more evident when those containers are constantly writing new files to the NFS mount, such as in an Artificial Intelligence/Machine Learning use case. FlexGroups were designed to handle massive amounts of file creations and can provide 2-6x the performance over a FlexVol in use cases where we’re constantly creating new files.

Stay tuned for some more information on how FlexGroups and Trident can bring even more capability to the table to AI/ML workloads. In the meantime, you can learn more about NetApp solutions for AI/ML here:

https://www.netapp.com/us/solutions/applications/ai-deep-learning.aspx

Behind the Scenes: Episode 99 – Databases as a Service, using Docker Containers

Welcome to the Episode 99, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

group-4-2016

This week on the podcast, we sync up with NetApp’s resident Oracle expert, Jeff Steiner (@tweetofsteiner), to discuss his work in containerizing databases.

Visit thePub to learn more about NetApp Docker Volume Plugin 17.07 and Jeff’s clone split contribution.

Want to know more about this topic? Register for NetApp Insight 2017 & attend Jeff Steiner’s session “Databases with NetApp ONTAP in Public and Private Clouds.” You can also reach Jeff via email at steiner@netapp.com or reach his blog at words.ofsteiner.com.

Additional Resources:

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

You can listen here:

Behind the Scenes: Episode 35 – Mesosphere DC/OS

ep35

Welcome to the Episode 35 version of the new series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

This week, we were fortunate to get the folks at Mesosphere over Skype to talk to us on the week of the release of DC/OS. Andrew was actually the driver of the scheduling here, and it was good timing.

This episode featured:

  • Kiersten Gaffney of Mesosphere (@kierstengaffney)
  • Aaron Williams of Mesosphere (@_arw)
  • Garrett Mueller, Technical Director at Netapp
  • Bryan Naylor, developer at NetApp

The official NetApp blog post on the episode can be found here: http://community.netapp.com/t5/Technology/Tech-ONTAP-Podcast-Episode-35-Mesosphere-DC-OS/ba-p/118579

Recording the podcast

This week, we had a combination of local people from NetApp, as well as Skype callers from Mesosphere.

The recording went pretty smoothly. Fun fact from last week – forgot to hit record for about 20 min into the OpenStack Summit podcast. True professionalism there.

DC/OS is ready to download and try today at https://dcos.io/. You can also find Andrew’s new Docker blog post at http://community.netapp.com/t5/Technology/Docker-Volumes-Using-NetApp-Storage/ba-p/108159 and a link for the new NetApp Docker plugin here: http://netapp.github.io/openstack/2016/04/19/announcing-the-netapp-docker-volume-plugin/index.html

Now, for the podcast!

TECH:: The NetApp Tech ONTAP Podcast and its Triumphant Return

straight-outta

A long time ago…

…in a galaxy far, far away, Nick Howell (aka @datacenterdude) decided it would be pretty awesome if NetApp had its very own podcast for people to listen to. He started it with Gabe Lowe. After 20 episodes, Gabe moved on to PernixData. Episode 23 introduced Pete Flecha to the fold.

Glenn Sizemore joined the fold after proving his chops on Microsoft, NetApp Shift and FlexPod.

I’m pretty sure they found Andrew Sullivan as a small child, being raised by a cult obsessed with Docker containers called “Tail of the Whale.” (But I could be making that up).

Full bios on those guys are below…

Why Podcast?

This was an effort to raise NetApp’s profile and social media cred, as well as maintain a running dialogue with customers and partners alike. The podcast would include information about NetApp, its products and feature subject matter experts in the storage industry to talk about a variety of topics, both standard and cutting edge.

The podcast was hosted on Nick’s site and operated in a sort of official/unofficial capacity. After a solid run of building a dedicated fan base, Nick moved on to another gig and the podcast was left in limbo.

micdrop

Would it be resurrected? Should it?

You don’t miss it until it’s gone

It turns out that the podcast was extremely popular. So popular, in fact, that there was a groundswell of support and furor coming from its listeners. People loved the podcast. They wanted it back. They needed it. So, NetApp delivered.

The phoenix rose from the ashes and the NetApp Communities Podcast was brought back as Tech ONTAP, which had the official blessing of NetApp as a company.

DarkPhoenix

Unlike before, the site was now hosted on the NetApp Communities. The official Twitter handle became @NetApp. And the show was hosted by a trio of Nerds With Attitudes. The first episode of the new podcast was posted on July 31, 2015.

podcast-black

Who hosts Tech ONTAP?

The podcast is currently hosted by 3 of the best and brightest at NetApp.

Pete Flecha @vPedroArrow

pete

Pete is the NetApp TME for Business Continuity and Disaster Recovery, including NetApp MetroCluster and SRM. He has his own blog called PedroArrow.com. Pete is the host of the Tech ONTAP podcast and is an absolute perfectionist, so he’ll spend hours at a time editing the podcast and doing re-takes, just to make sure it’s right. Pete has presented at VMworld and will be presenting a few sessions at NetApp Insight.

Here’s a list of Pete’s technical reports, if you’re interested:

When he’s not working, he’s getting absolutely destroyed at foosball by Dan Isaacs.*

*Paid for by Dan Isaacs

Glenn Sizemore @glnsize

glnsize

Glenn is a jack of all trades, but a master at all of them. FlexPod, anything Microsoft, HCI, cloud… and those aren’t just buzzy resume’ filler fluff. He ACTUALLY KNOWS THOSE THINGS.

Glenn is the co-host of the Tech ONTAP podcast and keeps Pete Flecha honest, as well as bickering with Andrew Sullivan. He’ll be presenting at NetApp Insight and likely knocking it out of the park.

Here’s a list of Glenn’s technical reports, if you’re interested:

Andrew Sullivan @andrew_ntap

sully 

Andrew Sullivan (aka Sully the Monster) is a VMware/DevOps/Docker/Automation TME for NetApp and has helped spearhead integration of containers onto NetApp storage and is active in the relationship between NetApp and ClusterHQ. Like Glenn, Andrew knows multiple technologies and can tell you more than you’d even want to know about them. However, you’d have issues hearing him, as he’s pretty soft spoken. Either that, or the beard acts as a muffler. He’s a co-host of the podcast and has every technology mastered – except microphones. He also writes up the podcast blog posts and has his own blog called practical-admin.com.

You can often find him locked in a hearty “discussion” with Glenn Sizemore from topics ranging from what the best use of containers are to if water is truly wet.

Here’s a list of Andrew’s technical reports and blogs, if you’re interested:

Who else?

Guests range from experts like Jay Goldfinch (cDOT TME)Dan Isaacs (AFF TME and Vaughn Stewart harasser) to EVO:RAIL master Eric Railine to FlexPod TME Melissa Palmer (@vmiss33) to myself. Coming weeks will feature even more top notch industry experts, so be sure to subscribe!

Each week, a new topic is covered at a deep, technical level. Additionally, the Tech ONTAP podcast gives daily wrap ups of conferences, such as VMworld and NetApp Insight. The fluff is kept to a minimum. The podcast’s intention is to educate and entertain.

At NetApp Insight, they’ll be interviewing NetApp employees, partners (including members of the esteemed NetApp A-Team) and customers daily to get their take on the conference. There will also be a Tech ONTAP booth, where you can meet the guys and ask them questions.

So if you can’t attend in person, you can at least keep up with what is going on from NetApp’s perspective.

Where can I find it?

The podcast is posted as a blog each week in the NetApp Communities. Taking a long road trip or flight? Download a few and take a listen. Have some downtime in between sessions at NetApp Insight? Listen to the podcast!

The first six episodes of the rebooted podcast are listed below:

The old podcast episodes are buried in New Mexico (where they found the Atari ET cartridges).

They are available on Soundcloud and iTunes for free download.

Be sure to also check out what sessions I will be doing at this blog post:

My NetApp Insight Sessions

TECH::Using NFS with Docker – Where does it fit in?

docker-nfs

NOTE: I wrote this blog nearly 2 years ago, so a lot has changed since then regarding Docker. 

Check out this newer blog on NFS/Docker for NetApp here:

Docker + NFS + FlexGroup volumes = Magic!

Also, check out how to Kerberize NFS in a container!

Securing NFS mounts in a Docker container

Recently, I wrote a post on DataCenterDude.com on how to use Docker to create the ultimate subtweet via a container running VNC and Firefox.

That got me thinking… as the NFS TME for NetApp, I have to consider where file services fit into newer technologies like Docker. What use cases might there be? Why would we use it?

In this blog, I will talk about what sorts of things you can do with NFS in Docker. I’ll say upfront that I’m a Docker rookie, so some of the things I do might seem kludgy to Docker experts. But I’m learning as I go and trying to document it here. 🙂

Can I store Docker images on NFS?

When you run a build of a Docker image, it gets stored in /var/lib/docker.

# ls
containers    graph         repositories-devicemapper  vfs
devicemapper  init          tmp                        volumes
execdriver    linkgraph.db  trust

However, that file size is limited. From the /etc/sysconfig/docker-storage file:

# By default, Docker uses a loopback-mounted sparse file in
# /var/lib/docker. The loopback makes it slower, and there are some
# restrictive defaults, such as 100GB max storage.

We can see that a sparse 100GB file is created in /var/lib/docker/devicemapper/devicemapper, as well as a 2GB metadata file. This seems to be where our images get stored.

# ls -lah /var/lib/docker/devicemapper/devicemapper
total 743M
drwx------. 2 root root 32 May 4 22:59 .
drwx------. 5 root root 50 May 4 22:59 ..
-rw-------. 1 root root 100G May 7 21:16 data
-rw-------. 1 root root 2.0G May 7 21:16 metadata

And we can see that there is actually 743MB in use.

So, we’re limited to 100GB and that storage is going to be fairly slow. However, the filesystems supported with Docker for this don’t include NFS. Instead, it uses block-based filesystems, as described here:

Comprehensive Overview of Storage Scalability in Docker

However, you could still use NFS to store the Docker image data and metadata. How?

Mount a NFS mount to /var/lib/docker!

When I stop Docker and mount the directory via NFS, I can see that there is nothing in my 1TB volume:

# service docker stop
# mount 10.228.225.142:/docker /var/lib/docker
# df -h | grep docker
10.228.225.142:/docker 973G 384K 973G 1% /var/lib/docker

When I start the Docker service, ~300MB is written to the mount and the folder structure is created:

# service docker start
Redirecting to /bin/systemctl start docker.service
# df -h | grep docker
10.228.225.142:/docker 973G 303M 973G 1% /var/lib/docker
# ls
containers devicemapper execdriver graph init linkgraph.db repositories-devicemapper 
tmp trust volumes

However, there is still a 100GB limited sparse file and we can see where the 300M is coming from:

# ls -lah
total 295M
drwx------ 2 root root 4.0K May 7 21:39 .
drwx------ 4 root root 4.0K May 7 21:39 ..
-rw------- 1 root root 100G May 7 21:39 data
-rw------- 1 root root 2.0G May 7 21:39 metadata

In this particular case, the benefits NFS brings here is external storage in case the host ever tanks or external storage for hosts that don’t have 100GB to spare. It would also be cool if that sparse file could be customized to grow past 100GB if we wanted it to. I actually posted a request for that to happen. Then they replied and I discovered I didn’t RTFM and it actually DOES exist as an option. 😉

So what could we use NFS for in Docker?

When you create a container, you are going to be fairly limited to the amount of space in that container. This is exacerbated when you run multiple containers on the same host. So, to get around that, you could mount a NFS share at the start of the container. With NFS, my storage limits are only what my storage provider dictates.

Another benefit of NFS with Docker? Access to a unified set of data across all containers.

If you’re using containers to do development, it would make sense that the containers all have access to the same code branches and repositories. What if you had an application that needed access to a shared Oracle database?

How do we do it?

One thing I’ve noticed while learning Docker is that the container OS is nothing like a virtual machine. These containers are essentially thin clients and are missing some functionality by design. From what I can tell, the goal is to have a client that can run applications but is not too heavyweight (for efficiency) and not terribly powerful in what can be done on the client (for security). For instance, the default for Linux-based OSes seems to leave systemd out of the equation. Additionally, these clients all start in unprivileged mode by default, so root is a bit limited in what it can and cannot do.

As a result, doing something as simple as configuring a NFS client can be a challenge.

Why not just use the -v option when running your container?

One approach to serving NFS to your Docker image would be to mount the NFS share to your Docker host and then run your docker images using the -v option to mount a volume. That eliminates the need to do anything wonky on the images and allows easier NFS access. However, I wanted to mount inside of a Docker container for 2 reasons:

  1. My own knowledge – what would it take? I learned a lot about Docker, containers and the Linux OS doing it this way.
  2. Cloud – If we’re mounting NFS on a Docker host, what happens if we want to use those images elsewhere in the world? We’d then have to mount the containers to those Docker hosts. Our end users would have to either run a script on the host or would need to know what to mount. I thought it might scale better to have it built in to the image itself and make it into a true “Platform as a Service” setup. Then again, maybe it *would* make more sense to do it via the -v option… I’m sure there could be use cases made for both.

Step 1: Configure your NFS server

In this example, I am going to use NFS running on clustered Data ONTAP 8.3 with a 1TB volume.

cluster::> vol show -vserver parisi -volume docker -fields size,junction-path,unix-permissions,security-style
vserver volume size security-style unix-permissions junction-path
------- ------ ---- -------------- ---------------- -------------
parisi  docker 1TB  unix           ---rwxr-xr-x     /docker

The NFS server will have NFSv3 and NFSv4.1 (with pNFS) enabled.

cluster::> nfs server show -vserver parisi -fields v3,v4.1,v4.1-pnfs,v4-id-domain
vserver v3      v4-id-domain             v4.1    v4.1-pnfs
------- ------- ------------------------ ------- ---------
parisi  enabled domain.win2k8.netapp.com enabled enabled

I’ll need an export policy and rule. For now, I’ll create one that’s wide open and apply it to the volume.

cluster::> export-policy rule show -vserver parisi -instance
                                    Vserver: parisi
                                Policy Name: default
                                 Rule Index: 1
                            Access Protocol: any
Client Match Hostname, IP Address, Netgroup, or Domain: 0.0.0.0/0
                             RO Access Rule: any
                             RW Access Rule: any
User ID To Which Anonymous Users Are Mapped: 65534
                   Superuser Security Types: any
               Honor SetUID Bits in SETATTR: true
                  Allow Creation of Devices: true

Step 2: Modify your Dockerfile

The Dockerfile is a configuration file that can be used to build custom Docker images. They are essentially startup scripts.

If you want to use NFS in your Docker container, the appropriate NFS utilities would need to be installed. In this example, I’ll be using CentOS 7. In this case, the Dockerfile would need to include a section with an install of nfs-utils:

# Install NFS tools
yum install -y nfs-utils

To run NFSv3, you have to ensure that the necessary ancillary processes (statd, nlm, etc) are running. Otherwise, you have to mount with nolock, which is not ideal:

# mount -o nfsvers=3 10.63.3.68:/docker /mnt
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

This requires systemd to be functioning properly in your Docker container image. This is no small feat, but is possible. Check out Running systemd within a Docker Container for info on how to do this with RHEL/Fedora/CentOS, or read further in this blog for some of the steps I had to take to get it working. There are containers out there that other people have built that run systemd, but I wanted to learn how to do this on my own and guarantee that I’d get exactly what I wanted from my image.

To run just NFSv4, all you need are the nfs-utils. No need to set up systemd.

# mount 10.63.3.68:/docker /mnt
# mount | grep docker
10.63.3.68:/docker on /mnt type nfs4 (rw,relatime,vers=4.0,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=172.17.0.21,local_lock=none,addr=10.63.3.68)

I essentially took the Dockerfile from the systemd post I mentioned, as well as this one for CentOS and modified them a bit (removed the rm -f entries, created directory for mount point, changed location of dbus.service, copied the dbus.service file to the location of the Dockerfile, installed nfs-utils and autofs).

I based it on this Dockerfile: https://github.com/maci0/docker-systemd-unpriv/

From what I can tell, mounting NFS in a Docker container requires privileged access and since there is currently no way to build in privileged mode, we can’t add mount command to the Dockerfile. So I’d need to run the mount command after the image is built. There are ways to run systemd in unprivileged mode, as documented in this other blog.

Using autofs!

On the client, we could also set up automounter to mount the NFS mounts when we need them, rather than mounting them and leaving them mounted. I cover automounting in NFS (for homedirs)with clustered Data ONTAP a bit in TR-4073 on page 160.

Doing this was substantially trickier, as I needed to install/start autofs, which requires privileged mode *and* systemd to be working properly. Plus, I had to do a few other tricky things.

This is what the /etc/auto.master file would look like in the container:

# cat /etc/auto.master
#
# Sample auto.master file
# This is a 'master' automounter map and it has the following format:
# mount-point [map-type[,format]:]map [options]
# For details of the format look at auto.master(5).
#
/misc /etc/auto.misc
#
# NOTE: mounts done from a hosts map will be mounted with the
# "nosuid" and "nodev" options unless the "suid" and "dev"
# options are explicitly given.
#
/net -hosts
#
# Include /etc/auto.master.d/*.autofs
# The included files must conform to the format of this file.
#
+dir:/etc/auto.master.d
#
# Include central master map if it can be found using
# nsswitch sources.
#
# Note that if there are entries for /net or /misc (as
# above) in the included master map any keys that are the
# same will not be seen as the first read key seen takes
# precedence.
#
+auto.master
/docker-nfs /etc/auto.misc --timeout=50

And the /etc/auto.misc file:

# cat /etc/auto.misc
docker -fstype=nfs4,minorversion=1,rw,nosuid,hard,tcp,timeo=60 10.228.225.142:/docker

The Dockerfile

Here’s my Dockerfile for a container that can run NFSv3 or NFSv4, with manual or automount.

FROM centos:centos7
ENV container docker
MAINTAINER: Justin Parisi "whyistheinternetbroken@gmail.com"
RUN yum -y update; yum clean all
RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs
RUN yum -y install nfs-utils; yum clean all
RUN systemctl mask dev-mqueue.mount dev-hugepages.mount \
 systemd-remount-fs.service sys-kernel-config.mount \
 sys-kernel-debug.mount sys-fs-fuse-connections.mount
RUN systemctl mask display-manager.service systemd-logind.service
RUN systemctl disable graphical.target; systemctl enable multi-user.target

# Copy the dbus.service file from systemd to location with Dockerfile
COPY dbus.service /usr/lib/systemd/system/dbus.service

VOLUME ["/sys/fs/cgroup"]
VOLUME ["/run"]

CMD ["/usr/lib/systemd/systemd"]

# Make mount point
RUN mkdir /docker-nfs

# Configure autofs
RUN yum install -y autofs
RUN echo "/docker-nfs /etc/auto.misc --timeout=50" >> /etc/auto.master

###### CONFIGURE THIS PORTION TO YOUR OWN SPECS ######
#RUN echo "docker -fstype=nfs4,minorversion=1,rw,nosuid,hard,tcp,timeo=60 10.228.225.142:/docker" >> /etc/auto.misc
######################################################


# Copy the shell script to finish setup
COPY configure-nfs.sh /configure-nfs.sh
RUN chmod 777 configure-nfs.sh
This was my shell script:
#!/bin/sh

# Start services
service rpcidmapd start
service rpcbind start
service autofs start

# Kill autofs pid and restart, because Linux
ps -ef | grep '/usr/sbin/automount' | awk '{print $2}' | xargs kill -9
service autofs start

The script had to live in the same directory as my Dockerfile. I also had to copy dbus.service to that same directory.

# cp /usr/lib/systemd/system/dbus.service /location_of_Dockerfile/dbus.service

Once I had the files in place, I built the image:

# docker build -t parisi/nfs-client .
Sending build context to Docker daemon 5.12 kB
Sending build context to Docker daemon
Step 0 : FROM centos:centos7
 ---> fd44297e2ddb
Step 1 : ENV container docker
 ---> Using cache
 ---> 2cbdf1a478bc
Step 2 : RUN yum -y update; yum clean all
 ---> Using cache
 ---> d4015989b039
Step 3 : RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs
 ---> Using cache
 ---> ec0fbd7641bb
Step 4 : RUN yum -y install nfs-utils; yum clean all
 ---> Using cache
 ---> 485fac2c1733
Step 5 : RUN systemctl mask dev-mqueue.mount dev-hugepages.mount systemd-remount-fs.service sys-kernel-config.mount sys-kernel-debug.mount sys-fs-fuse-connections.mount
 ---> Using cache
 ---> d7f44caa9d8f
Step 6 : RUN systemctl mask display-manager.service systemd-logind.service
 ---> Using cache
 ---> f86f635b6af7
Step 7 : RUN systemctl disable graphical.target; systemctl enable multi-user.target
 ---> Using cache
 ---> a4b7fed3b91d
Step 8 : COPY dbus.service /usr/lib/systemd/system/dbus.service
 ---> Using cache
 ---> f11fa8045437
Step 9 : VOLUME /sys/fs/cgroup
 ---> Using cache
 ---> e042e697636d
Step 10 : VOLUME /run
 ---> Using cache
 ---> 374fc2b247cb
Step 11 : CMD /usr/lib/systemd/systemd
 ---> Using cache
 ---> b797b045d6b7
Step 12 : RUN mkdir /docker-nfs
 ---> Using cache
 ---> 8228a9ca400d
Step 13 : RUN yum install -y autofs
 ---> Using cache
 ---> 01a64d46a737
Step 14 : RUN echo "/docker-nfs /etc/auto.misc --timeout=50" >> /etc/auto.master
 ---> Using cache
 ---> 78b63c672baf
Step 15 : RUN echo "docker -fstype=nfs4,minorversion=1,rw,nosuid,hard,tcp,timeo=60 10.228.225.142:/docker" >> /etc/auto.misc
 ---> Using cache
 ---> a2d99d3e1ba3
Step 16 : COPY configure-nfs.sh /configure-nfs.sh
 ---> Using cache
 ---> 70e71370149d
Step 17 : RUN chmod 777 configure-nfs.sh
 ---> Running in c1e24ab5b643
 ---> 4fb2c5942cbb

Removing intermediate container c1e24ab5b643
Successfully built 4fb2c5942cbb

# docker images parisi/nfs-client
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
parisi/nfs-client latest 4fb2c5942cbb 49 seconds ago 298.2 MB

Now I can start systemd for the container in privileged mode.

# docker run --privileged -d -v /sys/fs/cgroup:/sys/fs/cgroup:ro parisi/nfs-client sh -c "/usr/lib/systemd/systemd"
e157ecdf7269dce8178cffd54c74abef2a242cbfe522298ad410c292896991e8

# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e157ecdf7269 parisi/nfs-client:latest "sh -c /usr/lib/syst 11 seconds ago Up 10 seconds sleepy_pike

Then I can go into the container and kick off my script:

# docker exec -t -i e157ecdf7269dce8178cffd54c74abef2a242cbfe522298ad410c292896991e8 /bin/bash
[root@e157ecdf7269 /]# ./configure-nfs.sh
Redirecting to /bin/systemctl start rpcidmapd.service
Failed to issue method call: Unit rpcidmapd.service failed to load: No such file or directory.
Redirecting to /bin/systemctl start rpcbind.service
Redirecting to /bin/systemctl start autofs.service
kill: sending signal to 339 failed: No such process
Redirecting to /bin/systemctl start autofs.service

[root@e157ecdf7269 /]# service autofs status
Redirecting to /bin/systemctl status autofs.service
autofs.service - Automounts filesystems on demand
 Loaded: loaded (/usr/lib/systemd/system/autofs.service; disabled)
 Drop-In: /run/systemd/system/autofs.service.d
 └─00-docker.conf
 Active: active (running) since Mon 2015-05-11 21:03:24 UTC; 21s ago
 Process: 355 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid (code=exited, status=0/SUCCESS)
 Main PID: 357 (automount)
 CGroup: /system.slice/docker-e157ecdf7269dce8178cffd54c74abef2a242cbfe522298ad410c292896991e8.scope/system.slice/autofs.service
 └─357 /usr/sbin/automount --pid-file /run/autofs.pid

May 11 21:03:24 e157ecdf7269 systemd[1]: Starting Automounts filesystems on demand...
May 11 21:03:24 e157ecdf7269 automount[357]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
May 11 21:03:24 e157ecdf7269 systemd[1]: Started Automounts filesystems on demand.

Once the script runs, I can start testing my mounts!

First, let’s do NFSv3:

[root@e454fd0728bd /]# mount -o nfsvers=3 10.228.225.142:/docker /mnt
[root@e454fd0728bd /]# mount | grep mnt
10.228.225.142:/docker on /mnt type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.228.225.142,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.228.225.142)
[root@e454fd0728bd /]# cd /mnt
[root@e454fd0728bd mnt]# touch nfsv3file
[root@e454fd0728bd mnt]# ls -la
total 12
drwxrwxrwx 2 root root 4096 May 12 15:14 .
drwxr-xr-x 20 root root 4096 May 12 15:13 ..
-rw-r--r-- 1 root root 0 May 12 15:14 nfsv3file
-rw-r--r-- 1 root root 0 May 11 17:53 nfsv41
-rw-r--r-- 1 root root 0 May 11 18:46 nfsv41-file
drwxrwxrwx 11 root root 4096 May 12 15:05 .snapshot

Now, let’s try autofs!

When I log in, I can see that nothing is mounted:

[root@e157ecdf7269 /]# mount | grep docker
/dev/mapper/docker-253:1-50476183-e157ecdf7269dce8178cffd54c74abef2a242cbfe522298ad410c292896991e8 on / type ext4 (rw,relatime,discard,stripe=16,data=ordered)
/etc/auto.misc on /docker-nfs type autofs (rw,relatime,fd=19,pgrp=11664,timeout=50,minproto=5,maxproto=5,indirect)

Then I cd into my automount location and notice that my mount appears:

[root@e157ecdf7269 /]# cd /docker-nfs/docker
[root@e157ecdf7269 docker]# mount | grep docker
/dev/mapper/docker-253:1-50476183-e157ecdf7269dce8178cffd54c74abef2a242cbfe522298ad410c292896991e8 on / type ext4 (rw,relatime,discard,stripe=16,data=ordered)
/etc/auto.misc on /docker-nfs type autofs (rw,relatime,fd=19,pgrp=11664,timeout=50,minproto=5,maxproto=5,indirect)
10.228.225.142:/docker on /docker-nfs/docker type nfs4 (rw,nosuid,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=60,retrans=2,sec=sys,clientaddr=172.17.0.134,local_lock=none,addr=10.228.225.142)   <<<< there it is!

[root@e157ecdf7269 docker]# df -h | grep docker
/dev/mapper/docker-253:1-50476183-e157ecdf7269dce8178cffd54c74abef2a242cbfe522298ad410c292896991e8 9.8G 313M 8.9G 4% /
10.228.225.142:/docker 973G 1.7M 973G 1% /docker-nfs/docker

 What now?

Now that I’ve done all that work to create a Docker image, it’s time to share it to the world.

# docker login
Username (parisi):
Login Succeeded

# docker tag 7cb0f9093319 parisi/centos7-nfs-client-autofs

# docker push parisi/centos7-nfs-client-autofs

Do you really want to push to public registry? [Y/n]: Y
The push refers to a repository [docker.io/parisi/centos7-nfs-client-autofs] (len: 1)
Sending image list
Pushing repository docker.io/parisi/centos7-nfs-client-autofs (1 tags)
6941bfcbbfca: Image already pushed, skipping
41459f052977: Image already pushed, skipping
fd44297e2ddb: Image already pushed, skipping
78b3f0a9afb1: Image successfully pushed
ec8afb93938d: Image successfully pushed
3f5dcd409a0e: Image successfully pushed
231f18a54eee: Image successfully pushed
1bfe1aa6309e: Image successfully pushed
318c908bb3e7: Image successfully pushed
80f71e4c55e8: Image successfully pushed
1ef13e5d4686: Image successfully pushed
5d9999f99007: Image successfully pushed
ae28ae0477aa: Image successfully pushed
438342aef8e1: Image successfully pushed
1069a8beb629: Image successfully pushed
15692893daab: Image successfully pushed
8ce7a0621ca4: Image successfully pushed
4778761cf8bd: Image successfully pushed
7cb0f9093319: Image successfully pushed

Pushing tag for rev [7cb0f9093319] on {https://cdn-registry-1.docker.io/v1/repositories/parisi/centos7-nfs-client-autofs/tags/latest}

You can find the image repository here: https://registry.hub.docker.com/u/parisi/centos7-nfs-client-autofs/

And the files here: https://github.com/whyistheinternetbroken/docker-centos7-nfs-client-autofs

Also, check out this fantastic blog post by Andrew Sullivan on using NetApp snapshot technology to make Docker images persistent!

Installing Docker on a Mac? Check this out if you hit issues:

https://whyistheinternetbroken.wordpress.com/2016/03/25/docker-mac-install-fails/