Why Is the Internet Broken: Greatest Hits

When I started this site back in October of 2014, it was mainly to drive traffic to my NetApp Insight sessions -and it worked.

(By the way… stay tuned for a blog on this year’s new Insight sessions by yours truly. Now with more lab!)

As I continued writing, my goal was to keep creating content – don’t be the guy who just shows up during conference season.

blogfieldofdreams

So far, so good.

But since I create so much content, it gets hard to find for new visitors to this site, The WordPress archives/table of contents is lacking. So, what I’ve done is create my own table of contents of the top 5 most visited posts.

Top 5 Blogs (by number of visits)

TECH::Using NFS with Docker – Where does it fit in?

SMB1 Vulnerabilities: How do they affect NetApp’s Data ONTAP?

TECH::Become a clustered Data ONTAP CLI Ninja

ONTAP 9.1 is now generally available (GA)!

NetApp FlexGroup: An evolution of NAS

DataCenterDude

I also used to write for datacenterdude.com on occasion.

To read those, go to this link:

My DataCenterDude stuff

How else do I find stuff?

You can also search on the site or click through the archives, if you choose. Or, subscribe to the RSS feed. If you have questions or want to see something changed or added to the site, follow me on Twitter @NFSDudeAbides or comment on one of the posts here!

You can also email me at whyistheinternetbroken@gmail.com.

Behind the Scenes Episode 296: SAN Automation with NetApp and Broadcom

Welcome to the Episode 296, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, AJ Casamento (https://www.linkedin.com/in/aj-casamento-a44570) from Broadcom joins us to discuss NetApp SAN automation with Mike Peppers (@NTAPFLIGuy) and Madhu Pai of NetApp.

For more information:

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 296: SAN Automation with NetApp and Broadcom – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes Episode 295 – NetApp DataOps Toolkit

Welcome to the Episode 295, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, NetApp TME Michael Oglesby (https://www.linkedin.com/in/mboglesby) joins us to discuss data, AI/ML and the new DataOps Toolkit from NetApp!

For more information:

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 295: NetApp DataOps Toolkit – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes Episode 294 – Maintaining Business Operations through Disaster in ONTAP 9.9.1

Welcome to the Episode 294, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, we discuss how SnapMirror Business Continuity (SMBC) and MetroCluster over IP in ONTAP 9.9.1 helps maintain business operations through disasters, as well as when you’d want to use one over the other.

Joining us:

  • Manish Thakur, NetApp Product Manager, MetroCluster and SnapMirror Business Continuity (manish@netapp.com)
  • Cheryl George, NetApp Technical Marketing Engineer, MetroCluster and SnapMirror Business Continuity (cheryl.george@netapp.com)

For more information on MetroCluster over IP and SnapMirror Business Continuity:

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 294: Maintaining Business Operations through Disaster in ONTAP 9.9.1 – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes Episode 293 – NVMe: Disruptive Technology Brought to You Non-Disruptively by Brocade and NetApp

Welcome to the Episode 293, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, we discuss NVMe over Fibre Channel (NVMeFC) with experts from NetApp and Brocade, and how the protocol stacks up to the SCSI SAN protocols.

Joining us:

For more information on Brocade, NVMe and NetApp:

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 293: NVMe – Disruptive Technology Brought to You Non-Disruptively by Brocade and NetApp – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes Episode 292 – NetApp E-Series EF300

Welcome to the Episode 292, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, Mitch Blackburn (mitch.blackburn@netapp.com) joins us to discuss the E-Series EF300 and what’s new with the E-Series line.

For more information on the E-Series:

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 292: NetApp E-Series EF300 – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes Episode 291 – Introducing VMware Tanzu

Welcome to the Episode 291, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, we discuss VMware Tanzu, containers and how it all fits into the NetApp ecosystem with Myles Gray (@mylesagray) and Glenn Sizemore (@glnsize) of VMware and Chance Bingen (@CB8MyDataCenter) of NetApp.

For more information on VMware Tanzu and NetApp:

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 291: Introducing VMware Tanzu – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes Episode 290 – NetApp E-Series BeeGFS CSI Driver

Welcome to the Episode 290, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week we discuss NetApp E-Series support for BeeGFS and the new CSI driver for containers and Kubernetes deployments. Joining us are Eric Weber (eric.weber2@netapp.com) and Joe McCormick (@iamjoemccormick, https://www.linkedin.com/in/developedbyjoe/) from the NetApp E-Series engineering team.

For more information:

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 290: NetApp E-Series BeeGFS CSI Driver – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

Behind the Scenes Episode 289 – NetApp and Rubrik: Better Together

Welcome to the Episode 289, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, we discuss the latest news in the NetApp/Rubrik partnership and how Rubrik works with Ben Kendall (Alliances Technical Partner Manager, Americas at Rubrik, https://www.linkedin.com/in/ben-kendall-6436609/) and PF Guglielmi (Alliances Field CTO @pfguglielmi) of Rubrik and Chris Maino (Manager, Americas Solutions Architects, chris.maino@netapp.com) of NetApp.

For more information:

https://www.rubrik.com/en/partners/technology-partners/netapp

https://www.rubrik.com/en/company/newsroom/press-releases/21/rubrik-and-netapp-extend-partnership

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 289 – NetApp and Rubrik: Better Together – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss

How to see data transfer on an ONTAP cluster network

ONTAP clusters utilize a backend cluster network to allow multiple HA pairs to communicate and provide more scale for performance and capacity. This is done by allowing you to nondisruptively add new nodes (and, as a result, capacity and compute) into a cluster. Data will be accessible regardless of where you connect in the cluster. You can scale up to 24 nodes for NAS-only clusters, while being able to mix different HA pair types in the same cluster if you choose to offer different service levels for storage (such as performance tiers, capacity tiers, etc).

Network interfaces that serve data to clients live on physical ports on nodes and are floating/virtual IP addresses that can move to/from any node in the cluster. File systems for NAS are defined by Storage Virtual Machines (SVMs) and volumes. The SVMs own the IP addresses you would use to access data.

When using NAS (CIFS/SMB/NFS) for data access, you can connect to a data interface in the SVM that lives on any node in the cluster, regardless of where the data volume resides. The following graphic shows how that happens.

When you access a NAS volume on a data interface on the same node as the data volume, ONTAP can “cheat” a little and directly interact with that volume without having to do extra work.

If that data interface is on a different node than where the volume resides, then the NAS packet gets packaged up as a proprietary protocol and shipped over the cluster network backend to the node where the volume lives. This volume/node relationship is stored in an internal database in ONTAP so we always have a map to find volumes quickly. Once the NAS packet arrives on the destination node, it gets unpackaged, processed and then the response to the client goes back out the way it came.

Traversing the cluster network has a bit of a latency cost, however, as the packaging/unpackaging/traversal takes some time (more time than a local request). This manifests into slightly less performance for those workloads. The impact of that performance hit is negligible in most environments, but for latency-sensitive applications, there might be some noticeable performance degradation.

There are protocol features that help mitigate the remote I/O that can occur in a cluster, such as SMB node referrals and pNFS, but in scenarios where you can’t use either of those (SMB node referrals didn’t use Kerberos in earlier Windows versions; pNFS needs NFSv4.1 and later), then you’re going to likely have remote cluster traffic. As mentioned, in most cases this isn’t an issue, but it may be useful to have an easy way to find out if an ONTAP cluster is doing remote/cluster traffic.

Cluster level – Statistics show-periodic

To get a cluster-wide view if there is remote traffic on the cluster, you can use the advanced priv command “statistics show-periodic.” This command gives a wealth of information by default, such as:

  • CPU average/busy
  • Total ops/NFS ops/CIFS ops
  • FlexCache ops
  • Total data recieved/sent (Data and cluster network throughput)
  • Data received/sent (Data throughput only)
  • Cluster received/sent (Cluster throughput only)
  • Cluster busy % (how busy the cluster network is)
  • Disk reads/writes
  • Packets sent/received

We also have options to limit the intervals, define SVMs/vservers, etc.

::*> statistics show-periodic ?
[[-object] ] *Object
[ -instance ] *Instance
[ -counter ] *Counter
[ -preset ] *Preset
[ -node ] *Node
[ -vserver ] *Vserver
[ -interval ] *Interval in Seconds (default: 2)
[ -iterations ] *Number of Iterations (default: 0)
[ -summary {true|false} ] *Print Summary (default: true)
[ -filter ] *Filter Data

But for backend cluster traffic, we only care about a few of those, so we can filter the iterations for only what we want to view. In this case, I just want to look at the data sent/received and the cluster busy %.

::*> statistics show-periodic -counter total-recv|total-sent|data-recv|data-sent|cluster-recv|cluster-sent|cluster-busy

When I do that, I get a cleaner, easier to read capture. This is what it looks like when we have remote traffic. This is an NFSv4.1 workload without pNFS, using a mount wsize of 64K.

cluster1: cluster.cluster: 5/11/2021 14:01:49
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
    157MB   4.85MB    148MB   3.46MB      0%   8.76MB   1.39MB
    241MB   70.2MB    197MB   4.68MB      1%   43.1MB   65.5MB
    269MB    111MB    191MB   4.41MB      4%   78.1MB    107MB
    329MB   92.5MB    196MB   4.52MB      4%    133MB   88.0MB
    357MB    117MB    246MB   5.68MB      2%    111MB    111MB
    217MB   27.1MB    197MB   4.55MB      1%   20.3MB   22.5MB
    287MB   30.4MB    258MB   5.91MB      1%   28.7MB   24.5MB
    205MB   28.1MB    176MB   4.03MB      1%   28.9MB   24.1MB
cluster1: cluster.cluster: 5/11/2021 14:01:57
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
Minimums:
    157MB   4.85MB    148MB   3.46MB      0%   8.76MB   1.39MB
Averages for 8 samples:
    258MB   60.3MB    201MB   4.66MB      1%   56.5MB   55.7MB
Maximums:
    357MB    117MB    258MB   5.91MB      4%    133MB    111MB

As we can see, there is an average of 55.7MB sent and 56.5MB received over the cluster network each second; this accounts for an average of 1% of the available bandwidth, which means we have plenty of cluster network utilization left over.

When we look at the latency for this workload, this is what we see. (Using qos statistics latency show)

Policy Group            Latency
-------------------- ----------
-total-                364.00us
extreme-fixed          364.00us
-total-                619.00us
extreme-fixed          619.00us
-total-                490.00us
extreme-fixed          490.00us
-total-                409.00us
extreme-fixed          409.00us
-total-                422.00us
extreme-fixed          422.00us
-total-                474.00us
extreme-fixed          474.00us
-total-                412.00us
extreme-fixed          412.00us
-total-                372.00us
extreme-fixed          372.00us
-total-                475.00us
extreme-fixed          475.00us
-total-                436.00us
extreme-fixed          436.00us
-total-                474.00us
extreme-fixed          474.00us

This is what the cluster network looks like when I use pNFS for data locality:

cluster1: cluster.cluster: 5/11/2021 14:18:19
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
    208MB   6.24MB    206MB   4.76MB      0%   1.56MB   1.47MB
    214MB   5.37MB    213MB   4.85MB      0%    555KB    538KB
    214MB   6.27MB    213MB   4.80MB      0%   1.46MB   1.47MB
    219MB   5.95MB    219MB   5.40MB      0%    572KB    560KB
    318MB   8.91MB    317MB   7.44MB      0%   1.46MB   1.47MB
    203MB   5.16MB    203MB   4.62MB      0%    560KB    548KB
    205MB   6.09MB    204MB   4.64MB      0%   1.44MB   1.45MB
cluster1: cluster.cluster: 5/11/2021 14:18:26
    total    total     data     data cluster  cluster  cluster
     recv     sent     recv     sent    busy     recv     sent
 -------- -------- -------- -------- ------- -------- --------
Minimums:
    203MB   5.16MB    203MB   4.62MB      0%    555KB    538KB
Averages for 7 samples:
    226MB   6.28MB    225MB   5.22MB      0%   1.08MB   1.07MB
Maximums:
    318MB   8.91MB    317MB   7.44MB      0%   1.56MB   1.47MB

There is barely any cluster traffic other than the normal cluster operations. The “data” and “total” sent/received is nearly identical.

And the latency was an average of .1 ms lower.

Policy Group            Latency
-------------------- ----------

-total-                323.00us
extreme-fixed          323.00us
-total-                323.00us
extreme-fixed          323.00us
-total-                325.00us
extreme-fixed          325.00us
-total-                336.00us
extreme-fixed          336.00us
-total-                325.00us
extreme-fixed          325.00us
-total-                328.00us
extreme-fixed          328.00us
-total-                334.00us
extreme-fixed          334.00us
-total-                341.00us
extreme-fixed          341.00us
-total-                336.00us
extreme-fixed          336.00us
-total-                330.00us
extreme-fixed          330.00us

Try it out and see for yourself! If you have questions or comments, enter them below.

Behind the Scenes – Episode 288 – ONTAP System Manager 9.9.1

Welcome to the Episode 288, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”

2019-insight-design2-warhol-gophers

This week, NetApp Principal TME Chris Gebhardt (@chrisgeb), PM Aniket Singh (aniket.singh@netapp.com) and TME Yizhao Zhuang (yizhao.zhuang@netapp.com) join us to discuss the latest ONTAP System Manager 9.9.1 changes.

For more information about System Manager:

https://docs.netapp.com/us-en/ontap/

Podcast Transcriptions

If you want a searchable transcript of the episode, check it out here (just set expectations accordingly):

Episode 288 – ONTAP System Manager 9.9.1 – Transcript

Just use the search field to look for words you want to read more about. (For example, search for “storage”)

transcript.png

Be sure to give us feedback (or if you need a full text transcript – Gong does not support sharing those yet) on the transcription in the comments here or via podcast@netapp.com! If you have requests for other previous episode transcriptions, let me know!

Tech ONTAP Community

We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.

Tech ONTAP Podcast Community

techontap_banner2

Finding the Podcast

You can find this week’s episode here:

You can also find the Tech ONTAP Podcast on:

I also recently got asked how to leverage RSS for the podcast. You can do that here:

http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss