TECH::Data LIF best practices for NAS in cDOT 8.3

network_cabling1

NOTE: Some of the documents linked in this blog are in the process of being updated for 8.3, so they may contain older information about clustered Data ONTAP 8.2.x. Check back on occasion to these links to get the most up to date content.

Clustered Data ONTAP 8.3 allows storage administrators to provide the following benefits:

  • Seamless scale-out storage
  • Multiprotocol Unified Access (NFS, CIFS and SAN)
  • Non-disruptive operations

This is done by way of secure multi-tenant architecture with Storage Virtual Machines. This blog will cover logical interface (LIF) considerations for clustered Data ONTAP 8.3 as they pertain to NAS storage operations. These considerations will eventually be added to TR-4067. For a complete list of networking best practices, see TR-4182: Ethernet Storage Best Practices for cDOT Configurations.

Storage Virtual Machines (SVMs)

SVMs are logical storage containers that own storage resources such as flexible volumes, logical interfaces (LIFs), exports, CIFS shares, etc. Think of them as a storage “blade center” in your cluster. These SVMs share physical hardware resources in the cluster with one another, such as network ports/VLANs, aggregates with physical disk, CPU, RAM, switches, etc. As a result, load for SVMs can be balanced across a cluster for maximum performance and efficiency, or to leverage SaaS functionality, among other benefits.

Cluster considerations

A cluster can be comprised of several HA pairs of nodes (4 HA pairs/8 nodes with SAN, 12 HA pairs/24 nodes with NAS). Each node in the cluster has its own copy of a replicated database with the cluster and SVM configuration information. Additionally, each node has its own set of user space applications that handle cluster operations and node-specific caches, not to mention its own set of RAM, CPU, disks, etc. So while a cluster operates as a single entity, it does have the underlying concept of individualized components. As a result, it makes sense to take the physical hardware in a cluster under consideration when implementing and designing.

Data LIF considerations

Data LIFs can live on any physical port in a cluster that is added to a valid broadcast domain. These data LIFs are configured with SVM-aware routing mechanisms that allow for the correct pathing of ethernet traffic in a SVM, regardless of where a valid data LIF lives in the cluster. Prior to 8.3, SVMs routed at a node level, so traffic could only travel via the node that owned a data LIF. In cDOT 8.3, traffic will route from the data LIF even if it is a non-local path.

However, despite this enhancement in cDOT 8.3, it is still worth considering the original best practice recommendation for data LIFs participating in NAS operations…

One data LIF per node, per SVM

With the introduction of IP Spaces in clustered Data ONTAP, this recommendation is more of a reality, as storage administrators no longer have to use unique IP addresses in the cluster for SVMs. With IP Spaces, IP addresses can be duplicated in the cluster on a per-SVM basis to allow for true secure multi-tenancy architecture. For more information on IP Spaces, see TR-4182.

Thus, if using a 24 node cluster, using 24 data LIFs per SVM would be ideal for the following reasons:

  • Ability to leverage data locality features – Features such as NFS referrals, CIFS auto location and pNFS ensure data locality for NAS traffic regardless of where the volumes live in a cluster. These help balance load better, but also make use of local caches and fastpath mechanisms for NAS traffic. For more information, see TR-4067.
  • Ability to reduce cluster network traffic – While cluster network traffic is generally not an issue (you’re more likely to peg the CPU or disk before you saturate a 10gb network), it is better to limit the amount of traffic on a cluster network as much as possible.
  • Ability to ensure data locality in the event of a volume move – If you move a volume to another node, you can ensure you still have a local path to the data if every node has a data LIF for the SVM.
  • Ability to spread the load out across nodes and leverage all the available hardware (CPU, RAM, etc) – If you load up all your NAS traffic on one node via one data LIF, you aren’t realizing the value of the other nodes in the cluster. Spreading network traffic ensures all available physical entities are being utilized. Why pay for hardware you aren’t using?
  • Ability to balance network connections across multiple cluster nodes – Clusters are single entities, as are SVMs. But they do have underlying hardware that have their own maximums, such as number of connections, etc. For info on hardware maximums in cDOT, see the configuration information for your version of ONTAP.
  • Ability to reduce impact of storage failover/givebacks (SFO). – Fewer clients impacted when SFO events happen, whether they are planned or unplanned.
  • Ability to leverage features such as on-box DNS – On-box DNS allows data LIFs to act as DNS servers and honor forwarded zone requests. Once a zone request is received, the cluster will determine the ideal node to service that request based on that node’s CPU and throughput, providing intelligent DNS load balancing (as opposed to round robin DNS, which is a serial process). For more information regarding on-box DNS (and how to configure it), see TR-4182 and TR-4073.

Keep in mind that the above are merely recommendations and not requirements unless using data locality features such as pNFS.

Any questions? Comments? Suggestions for blog topics? Feel free to comment or contact me on Twitter.

Advertisements

12 thoughts on “TECH::Data LIF best practices for NAS in cDOT 8.3

  1. I am curious what you mean by… “Prior to 8.3, SVMs routed at a node level, so traffic could only travel via the node that owned a data LIF. In cDOT 8.3, traffic will route from the data LIF even if it is a non-local path.”

    Did Nodes ever own LIFS? I thought Nodes owned Ports and LIFS were owned by an SVM and could be moved to any (appropriate) PORT in the cluster. also curious what you mean by traffic could only travel by the node that owned the data LIF, but in 8.3 traffic will route from the data LIF even if its is a non-local path. I’m having a hard time wrapping my head around that.

    That doesn’t seem correct to me. I thought direct and indirect I/O were part of Clustered ONTAP. if the request came in on a LIF but the volume was on a different node the I/O crosses the Cluster network and the response traverses it again to exit via the same LIF it came in on. I’m speaking of NAS traffic. SMB 1/2/2.1 and even 3, without doing autolocation. As far as i know the request coming in on LIF1 but exiting on LIF2 won’t/can’t happen.

    Like

    • The phrasing was a bit off. I was referring to auth requests. In 8.2.x and prior, if you issued an auth request on a node with a data LIF, that request would travel on that LIF. If you issued an auth request on a node with no data LIF in the SVM, the request would fail, as it was unable to route out of the node. In 8.3, auth requests can route anywhere in the cluster in SVMs if a routeable data LIF is present on any node.

      Nodes never “own” LIFs, as you state. But they do live on physical ports owned by physical nodes, which have individual processes.

      Regular NAS traffic (NFS, SMB) will arrive on a data LIF and leverage the owning node’s processes.

      Hope that helps.

      Like

  2. That makes a lot more sense to me. I thought you were talking about Data LIFS, not Mgmt LIFs and thats what confused me. Thanks for clarifying.

    Like

  3. So, For 8.3.X – system that runs up to 20 VMware NFS Datastores… Can I use single LIF only for all of them, Or should I, from some reason, Have separate LIF for each datastore?
    No plans in the near or far future to scale out the cluster.
    thanks,
    Itay

    Like

  4. From TR-4333 (April 2016) :
    Best Practice
    For each NFS datastore, provision a LIF for the SVM on each node.

    I have about 80 NFS datastores to migrate from 7-mode to cluster mode. What is the real best practice from your point of view ? Thanks.

    Like

    • As far as I know, that best practice has been changed to a data LIF per node, per SVM, regardless of number of datastores. So no more 1:1 LIF to datastore needed. Next release of that TR should cover it.

      Like

  5. Pingback: Why Is the Internet Broken: Greatest Hits | Why Is The Internet Broken?

  6. Pingback: Spreading the love: Load balancing NAS connections in ONTAP | Why Is The Internet Broken?

  7. Any idea on next release date on that TR? I want to host 100 NFS vols on 4-6 nodes and then be able to vol move them individually to other aggrs(nodes) without having to change their mount IP. I was about to set up 1 mount IP per lif per datastore instead

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s