Why Is the Internet Broken: Greatest Hits

When I started this site back in October of 2014, it was mainly to drive traffic to my NetApp Insight sessions -and it worked.

(By the way… stay tuned for a blog on this year’s new Insight sessions by yours truly. Now with more lab!)

As I continued writing, my goal was to keep creating content – don’t be the guy who just shows up during conference season.

blogfieldofdreams

So far, so good.

But since I create so much content, it gets hard to find for new visitors to this site, The WordPress archives/table of contents is lacking. So, what I’ve done is create my own table of contents of the top 5 most visited posts.

Top 5 Blogs (by number of visits)

TECH::Using NFS with Docker – Where does it fit in?

SMB1 Vulnerabilities: How do they affect NetApp’s Data ONTAP?

TECH::Become a clustered Data ONTAP CLI Ninja

ONTAP 9.1 is now generally available (GA)!

NetApp FlexGroup: An evolution of NAS

DataCenterDude

I also used to write for datacenterdude.com on occasion.

To read those, go to this link:

My DataCenterDude stuff

How else do I find stuff?

You can also search on the site or click through the archives, if you choose. Or, subscribe to the RSS feed. If you have questions or want to see something changed or added to the site, follow me on Twitter @NFSDudeAbides or comment on one of the posts here!

You can also email me at whyistheinternetbroken@gmail.com.

Advertisements

Behind the Scenes: Episode 128 – OnCommand Unified Manager 7.3

Welcome to the Episode 128, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, Yossi Weihs (https://www.linkedin.com/in/yossiw/) and Philip Bachman (@pgbachman) join us to talk about the latest OnCommand Unified Manager release and make a compelling case around why everyone should be installing it.

If you’re interested in learning about OCUM 7.2, check out the previous podcast found here:

Behind the Scenes: Episode 83 – OnCommand Unified Manager 7.2

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Removal of IP Fastpath in ONTAP 9.2

ONTAP 9.3 is already GA as of December, but there was a pretty significant change to how networking operates in ONTAP 9.2. However, this change impacts any release after 9.1, so it’s good to bring up.

I never really touched on it before, because we hadn’t seen any issues with it. However, we’re now seeing occasional problems resulting from that change, so I want to detail it all here.

flash.jpg

ONTAP 9.2 has modernized its networking stack to take advantage of new features and capabilities introduced by FreeBSD. As a result, one of the networking features that ONTAP used went away. This was known as “IP Fastpath.” (This also resulted in the removal of routing groups.)

What is IP Fastpath?

From the product docs:
https://library.netapp.com/ecmdocs/ECMP1368834/html/GUID-8276014A-16EB-4902-9EDC-868C5292381B.html

IP Fast path is an alternative routing mechanism to the routing table. In fast path, the responses to incoming network traffic are sent back by using the same interface as the incoming traffic. By avoiding the routing table lookup, fast path provides a quick access to data.

If fast path is enabled on an interface group and a physical interface in that group receives an incoming request, the same physical interface might not send a response to the request. Instead, any other physical interface in an interface group can send the response.

How fast path works with NFS/UDP

NFS/UDP traffic uses fast path only when sending a reply to a request. The reply packet is sent out on the same network interface that received the request packet.For example, a storage system named toaster uses the toaster-e1 interface to send reply packets in response to NFS/UDP requests received on the toaster-e1 interface.Fast path is used only in NFS/UDP. However, fast path is not used in other UDP-based NFS services such as portmapper, mountd, and nlm.

How fast path works with TCP

In a TCP connection, fast path is disabled on the third retransmission and the consecutive retransmissions of the same data packet. If Data ONTAP initiates a connection, Data ONTAP can use fast path on every TCP packet transmitted, except the first SYN packet. The network interface that is used to transmit a packet is the same interface that received the last packet.

IP Fastpath was not without faults, however. If you used asymmetric routing, you could run into issues:

Fast path not compatible with asymmetric routing

In a symmetric network, the destination MAC address of the response packet is that of the router that forwarded the incoming packet. However, in asymmetric networks, the router that forwards packets to your storage system is not the router that forwards packets sent by the storage system. Therefore, in asymmetric networks, you must disable fast path.

HSRP also could present problems.

http://dustydev.blogspot.com/2012/10/io-fastpathing-on-netapp-controller.html

Also, if you used certain types of NIC teaming on clients, you could also see problems because the NIC team would send the MAC address of the underlying NIC and ONTAP would try to send it to that same MAC, but when it hits the NIC team, the NIC that gets the request might not be the same one that sent it. For example:

https://community.netapp.com/t5/Backup-and-Restore-Discussions/HP-Network-Teaming-Performance-Issue-Connecting-to-Filer/td-p/52335

In fact, if there were occasions where fastpath could be disabled on interfaces if the connection determined there was a loss. Then the fall back was to use the routing table. If the routing tables were bad or incorrectly configured, you’d experience an outage. IP fastpath actually would mask latent routing issues.

(The takeaway you should have from this is that removing IP fastpath is actually a good thing.)

Impact of IP Fastpath Removal

There’s a really good KB article on the impact that can be found here:

https://kb.netapp.com/app/answers/answer_view/a_id/1072895

ONTAP 9.2 replaced fastpath with always-on route caching to maintain the same level of performance without the unforeseen headaches that could occur with asymmetric routing. However, if your network design relied on IP fastpath to route traffic and bypass bad routing tables (including dynamic and static routes), then upgrading ONTAP to 9.2 would make those issues come to light.

Or maybe you wanted to purposely ignore routing tables, such as what went on in this article:

https://community.netapp.com/t5/Data-ONTAP-Discussions/NetApp-Ontap-9-2-Upgrade-review-your-network-first/td-p/136657

Things to check before upgrading to ONTAP 9.2 (and later)

  • How is your routing table currently configured in the SVM? Are you using 0.0.0.0/0 default routes? (network route show from the CLI)
  • Compare your connected clients (network connections active show) to your current routes. Do all clients have a route to the appropriate SVMs?
  • Are there multiple routes with the same routing metric? If so, any of those routes may be chosen. Will any of them break connectivity?

In most cases, this change won’t cause an issue, but if you do see intermittent network connectivity, network performance degradation or other network related issues after upgrading to ONTAP 9.2, look at this change first and open a support case to get assistance.

If you have any questions, feel free to leave them in the comments and I’ll get them answered!

Behind the Scenes: Episode 127 – NetApp United

Welcome to the Episode 127, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, NetApp United is looking for new members!

Image result for netapp united

We bring the head of the program, Petya Stefanova (@PeytStefanova) to give us a rundown, as well as some of the current NetApp United members to tell us about their experiences.

Included in the discussion:

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 126 – Komprise

Welcome to the Episode 126, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we bring in Komprise (@Komprise) CEO Kumar Goswami (@KumarKGoswami) to chat about data management and how their software helps get the most out of your NetApp storage systems!

komprise

For more information about Komprise, check out komprise.com!

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 125 – Machine Learning and ONTAP

Welcome to the Episode 125, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we invited Senior Technical Director Santosh Rao in to discuss his blog series on Machine Learning in ONTAP, as well as how NetApp is helping power the latest and greatest in data analytics technology.

You can find Santosh’s blogs here:

https://blog.netapp.com/addressing-ai-data-lifecycle-challenges-with-data-fabric/

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 124 – SnapCenter 4.0

Welcome to the Episode 124, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we break down the latest and greatest features available in SnapCenter 4.0 with SnapCenter Product Manager, Anand Ranganathan (https://www.linkedin.com/in/anandrang/)!

You can also check out the official SnapCenter 4.0 blog here:

https://blog.netapp.com/snapcenter-4-0-whats-new-why-you-should-deploy-it/

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 123 – StorageGRID 11

Welcome to the Episode 123, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we bring in Tech ONTAP Podcast vet Duncan Moore (@NCDunc) to give us a rundown of StorageGRID 11, NetApp’s object storage solution!

Learn more about StorageGRID 11 here:

https://blog.netapp.com/storagegrid-11-0-the-best-cloud-integrated-object-storage-period/

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

Behind the Scenes: Episode 122 – Infrastructure as a Service

Welcome to the Episode 122, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

tot-gopher

This week on the podcast, we chat with a couple of NetApp’s cloud guys – Mark Beaupre (@Mark_Beaupre) and John Fullbright – about the latest way to consume ONTAP storage. Join us as we talk about Infrastructure as a Service in AWS and Azure and find out where it’s heading and when you can get access.

Finding the Podcast

The podcast is all finished and up for listening. You can find it on iTunes or SoundCloud or by going to techontappodcast.com.

This week’s episode is here:

Also, if you don’t like using iTunes or SoundCloud, we just added the podcast to Stitcher.

http://www.stitcher.com/podcast/tech-ontap-podcast?refid=stpr

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Our YouTube channel (episodes uploaded sporadically) is here:

ONTAP 9.3 is now GA!

ONTAP 9 is on a new cadence model, which brings a new release every 6 months.

Today, ONTAP 9.3GA is available here!

http://mysupport.netapp.com/NOW/download/software/ontap/9.3

ONTAP 9.3 was announced at NetApp Insight 2017 in Las Vegas and was covered at a high level by Jeff Baxter in the following blog:

Announcing NetApp ONTAP 9.3: The Next Step in Modernizing Your Data Management

Jeff has a follow-up infographic here:

https://blog.netapp.com/10-good-reasons-to-upgrade-to-ontap-9-3-infographic/

I also did a brief video summary here:

We also did a podcast with ONTAP Chief Evangelist Jeff Baxter (@baxontap) and ONTAP SVP Octavian Tanase (@octav) here:

For info on what GA means, see:

http://mysupport.netapp.com/NOW/products/ontap_releasemodel/

Also, check out the documentation center:

docs.netapp.com/ontap-9/index.jsp

The general theme around ONTAP 9.3 is modernization of the data center. Here’s a high level list of features, with more detail on some of them later in this blog.

Security enhancements

Simplicity innovations

  • MongoDB support added to application provisioning
  • Simplified data protection flows in System Manager
  • Guided cluster setup and expansion
  • Adaptive QoS

Performance and efficiency improvements

  • Up to 30% performance improvement for specific workloads via WAFL improvements, parallelization and flash optimizations
  • Automatic schedules for deduplication
  • Backgroup inline aggregate deduplication (AFF only; automatic schedule only)

NetApp FlexGroup volume features

This is covered in more detail in What’s New for NetApp FlexGroup Volumes in ONTAP 9.3?

  • Qtrees
  • Antivirus
  • Volume autogrow
  • SnapVault/Unified SnapMirror
  • SMB Change/notify
  • QoS Maximums
  • Improved automated load balancing logic

Data Fabric additions

  • SolidFire to ONTAP SnapMirror
  • MetroCluster over IP

Now, let’s look at a few of the features in a bit more detail. If you have things you want covered more, leave a comment.

Multifactor Authentication (MFA)

Traditionally, to log in to an ONTAP system as an admin, all you needed was a username and password and you’d get root-level access to all storage virtual machines in a cluster. If you’re the benevolent storage admin, that’s great! If you’re a hostile actor, great!* (*unless you’re the benevolent storage admin… then, not so great)

ONTAP 9.3 introduces the ability to configure an external Identity Provider (IdP) server to interact with OnCommand System Manager and Unified Manager to require a key to be passed in addition to a username and password. Initial support for IdP will include Microsoft Active Directory Federation Services and Shibboleth.

MFA

For the command line, the multifactor portion would be passed by way of SSH keys currently. We cover MFA in the following Tech ONTAP podcast:

SnapLock Enhancements

SnapLock is a NetApp ONTAP feature that provides data compliance for businesses that need to preserve data for regulatory reasons, such as HIPAA standards (SnapLock compliance) or for internal requirements, such as needing to preserve records (SnapLock enterprise).

ONTAP 9.3 provides a few enhancements to SnapLock, including one that isn’t available from any storage provider currently.

legal-hold.png

Legal hold is useful in the event that a court has ordered specific documents to be preserved for an ongoing case or investigation. This can be applied to multiple files and remains in effect until you choose to remove it.

event-based

Event-based retention allows storage administrators to set protections on data based on defined events, such as an employee leaving the company (to avoid disgruntled deletions), or for insurance use cases (such as death of a policy holder).

vol-append.png

Volume append mode is the SnapLock feature I alluded to, where no one else can currently accomplish this. Essentially, it’s for media workloads (audio and video) and will write-protect the portion of the files that have already been streamed and allow appending to those files after they’ve been protected. It’s kind of like having a CD-R on  your storage system.

Performance improvements

improve-perf

Every release of ONTAP strives to improve performance in some way. ONTAP 9.3 introduces performance enhancements (mostly for SAN)/block via the following changes:

  • Read latency reductions via WAFL optimizations for All Flash FAS SAN (block) systems
  • Better parallelization for all workloads on mid-range and high-end systems (FAS and AFF) to deliver more throughput/IOPS at lower latencies
  • Parallelization of the iSCSI layer to allow iSCSI to use more cores (best results on 20 core or higher systems)

The following graphs show some examples of that performance improvement versus ONTAP 9.2.

a700-fcp

a700-iscsi

Adaptive Quality of Service (QoS)

Adaptive QoS is a way for storage administrators to allow ONTAP to manage the number of IOPS per TB of volume space without the need to intervene. You simply set a service level class and let ONTAP control the rest.

The graphic below shows how it works.

adaptive-qos

We cover QoS minimums and performance enhancements in the following Tech ONTAP podcast:

MetroCluster over IP

MetroCluster is a way for clusters to operate in a high availability manner over long distances. (hundreds of kilometers) Traditionally, MetroCluster has been done over fiber channel networks due to low latency requirements needed to guarantee writes can be committed to both sites.

However, now that IP networks are getting more robust, ONTAP is able to support MetroCluster over IP, which provides the following benefits:

  • Reduced CapEx and OpEx (no more dedicated fiber channel networks, cards, bridges)
  • Simplicty of management (use existing IP networks)

mcc-ip.png

The ONTAP 9.3 release is going to be a limited release for this feature, with the following caveats:

  • A700, FAS9000 only
  • 100km limit
  • Dedicated ISL with extended VLAN currently required
  • 1 iWARP card per node

We cover MetroCluster over IP in this podcast:

SolidFire to ONTAP SnapMirror

A few years back, the concept of a data fabric (where all of your data can be moved anywhere with the click of a button) was introduced.

That vision continued this year with the inclusion of SnapMirror from SolidFire (and NetApp HCI systems) to ONTAP.

sf-snapmirror.png

ONTAP 9.3 will allow storage administrators to implement a disaster recovery plan for their SolidFire systems.

This includes the following:

  • Baseline and incremental replication using NetApp SnapMirror from SolidFire to ONTAP
  • Failover storage to ONTAP for disaster recovery
  • Failback storage from ONTAP to SolidFire
    • Only for LUNs replicated from SolidFire
    • Replication from ONTAP to SolidFire only for failback

That covers a deeper look at some of the new ONTAP 9.3 features. Feel free to comment if you want to learn more about these features, or any not listed in the overview.

FlexGroup Technical Reports Updated for ONTAP 9.3

fg-diagram

The latest updates for NetApp FlexGroup volumes for ONTAP 9.3 are available in the following Technical Reports:

Check it out and comment if you have a question!

Also check out previous blogs on FlexGroup volumes:

NetApp FlexGroup: Crazy fast

Tech ONTAP Podcast: Now powered by NetApp FlexGroup volumes!

NetApp FlexGroup: An evolution of NAS

And the lightboard video: