ONTAP 9.3 is already GA as of December, but there was a pretty significant change to how networking operates in ONTAP 9.2. However, this change impacts any release after 9.1, so it’s good to bring up.
I never really touched on it before, because we hadn’t seen any issues with it. However, we’re now seeing occasional problems resulting from that change, so I want to detail it all here.
ONTAP 9.2 has modernized its networking stack to take advantage of new features and capabilities introduced by FreeBSD. As a result, one of the networking features that ONTAP used went away. This was known as “IP Fastpath.” (This also resulted in the removal of routing groups.)
What is IP Fastpath?
From the product docs:
https://library.netapp.com/ecmdocs/ECMP1368834/html/GUID-8276014A-16EB-4902-9EDC-868C5292381B.html
IP Fast path is an alternative routing mechanism to the routing table. In fast path, the responses to incoming network traffic are sent back by using the same interface as the incoming traffic. By avoiding the routing table lookup, fast path provides a quick access to data.
If fast path is enabled on an interface group and a physical interface in that group receives an incoming request, the same physical interface might not send a response to the request. Instead, any other physical interface in an interface group can send the response.
How fast path works with NFS/UDP
NFS/UDP traffic uses fast path only when sending a reply to a request. The reply packet is sent out on the same network interface that received the request packet.For example, a storage system named toaster uses the toaster-e1 interface to send reply packets in response to NFS/UDP requests received on the toaster-e1 interface.Fast path is used only in NFS/UDP. However, fast path is not used in other UDP-based NFS services such as portmapper, mountd, and nlm.
How fast path works with TCP
In a TCP connection, fast path is disabled on the third retransmission and the consecutive retransmissions of the same data packet. If Data ONTAP initiates a connection, Data ONTAP can use fast path on every TCP packet transmitted, except the first SYN packet. The network interface that is used to transmit a packet is the same interface that received the last packet.
IP Fastpath was not without faults, however. If you used asymmetric routing, you could run into issues:
Fast path not compatible with asymmetric routing
In a symmetric network, the destination MAC address of the response packet is that of the router that forwarded the incoming packet. However, in asymmetric networks, the router that forwards packets to your storage system is not the router that forwards packets sent by the storage system. Therefore, in asymmetric networks, you must disable fast path.
HSRP also could present problems.
http://dustydev.blogspot.com/2012/10/io-fastpathing-on-netapp-controller.html
Also, if you used certain types of NIC teaming on clients, you could also see problems because the NIC team would send the MAC address of the underlying NIC and ONTAP would try to send it to that same MAC, but when it hits the NIC team, the NIC that gets the request might not be the same one that sent it. For example:
In fact, if there were occasions where fastpath could be disabled on interfaces if the connection determined there was a loss. Then the fall back was to use the routing table. If the routing tables were bad or incorrectly configured, you’d experience an outage. IP fastpath actually would mask latent routing issues.
(The takeaway you should have from this is that removing IP fastpath is actually a good thing.)
Impact of IP Fastpath Removal
There’s a really good KB article on the impact that can be found here:
https://kb.netapp.com/app/answers/answer_view/a_id/1072895
ONTAP 9.2 replaced fastpath with always-on route caching to maintain the same level of performance without the unforeseen headaches that could occur with asymmetric routing. However, if your network design relied on IP fastpath to route traffic and bypass bad routing tables (including dynamic and static routes), then upgrading ONTAP to 9.2 would make those issues come to light.
Or maybe you wanted to purposely ignore routing tables, such as what went on in this article:
Things to check before upgrading to ONTAP 9.2 (and later)
- How is your routing table currently configured in the SVM? Are you using 0.0.0.0/0 default routes? (network route show from the CLI)
- Compare your connected clients (network connections active show) to your current routes. Do all clients have a route to the appropriate SVMs?
- Are there multiple routes with the same routing metric? If so, any of those routes may be chosen. Will any of them break connectivity?
In most cases, this change won’t cause an issue, but if you do see intermittent network connectivity, network performance degradation or other network related issues after upgrading to ONTAP 9.2, look at this change first and open a support case to get assistance.
If you have any questions, feel free to leave them in the comments and I’ll get them answered!
Why Is The Internet Broken? 
Hi Justin,
We simply used fastpath to ignore the routing table, not because it was bad but because some of our application people like the functionality.
Would appreciating you correcting your wording when linking to my article.
Thanks,
Mark
LikeLike
Sure thing.
LikeLike
Hi Justin.
it’s not related to routing and this article. Since SMB multichannel & NFS trunking not available in ONTAP yet and many people still using LACP, I’ve wrote code which can tell what path will be selected depending on source & destination IP with SuperFastHash load balancing algorithm in ONTAP.
github.com qdrddr/ontap-lacp
LikeLike
“If fast path is enabled on an interface group and a physical interface in that group receives an incoming request, the same physical interface might not send a response to the request. Instead, any other physical interface in an interface group can send the response.”
This is a perfect example of poorly written, confusing documentation 🙂
It should say something along the lines of “Outside of IP fast path, an outgoing packet (response) might not use the same physical interface that the request was received on. Enabling IP fast path will make the response go out the same port used to receive the request.”
LikeLike