Welcome to the Episode 349, part of the continuing series called “Behind the Scenes of the NetApp Tech ONTAP Podcast.”
This week on the Tech ONTAP Podcast, Chris Hurley (@AverageGuyX) and Elliott Ecton (elliott.ecton@netapp.com) join us to discuss the latest FlexCache updates in ONTAP 9.12.1, as well as the latest information on the set of storage efficiency features in NetApp ONTAP.
We also now have a presence on the NetApp Communities page. You can subscribe there to get emails when we have new episodes.
Finding the Podcast
You can find this week’s episode here:
I’ve also resurrected the YouTube playlist. You can find this week’s episode here:
You can also find the Tech ONTAP Podcast on:
I also recently got asked how to leverage RSS for the podcast. You can do that here:
http://feeds.soundcloud.com/users/soundcloud:users:164421460/sounds.rss
Transcription
The following transcript was generated using Descript’s speech to text service and then further edited. As it is AI generated, YMMV.
Episode 349: NetApp FlexCache and Storage Efficiency Updates for ONTAP 9.12.1 – Transcript
Episode 349
===
Justin Parisi: This week on the Tech ONTAP podcast, we cover storage efficiency as well as the new FlexCache updates for ONTAP 9.12.1
Podcast intro: [podcast intro music]
Justin Parisi: Hello and welcome to the Tech ONTAP podcast. My name is Justin Parisi. I’m here in the basement of my house and with me today, I have a couple of special guests to talk to us about some new functionality in ONTAP 9.12.1 as well as some stuff that we haven’t really covered before for ONTAP. So with us today we have Elliot Ecton.
Elliot, what do you do here at NetApp and how do we reach you?
Elliott Ecton: Hey, Justin, I’m a technical marketing engineer for NAS protocols. Kind of took over, slid into your old spot a little bit. If you wanna get ahold of me, you can email me at elliott.ecton@netapp.com.
Justin Parisi: All right. Also with us today, Chris Hurley. Chris, what do you do here at NetApp and how do we reach you?
Chris Hurley: Hey Justin. So I am a senior Technical Product Manager for ONTAP here at NetApp, and you can reach me at, on Twitter @AverageGuyX and of course my NetApp email. So Christopher.Hurley@netapp.com.
Justin Parisi: All right, excellent. So we are here to talk about some NAS related stuff.
Specifically we’ll talk about some FlexCache advancements. But let’s start off with storage efficiency. We’ve not really gone into a lot of detail about some of the newer stuff that’s been in ONTAP in the last few releases. So Chris, what have we done in the last few releases with, with storage efficiency and when did we start doing it?
Chris Hurley: Yeah. So in ONTAP 9.8, we introduce something called TSSE. So temperature sensitive storage, efficiency. So what that means is, your cold data, we’re gonna take and recompress it with a larger compression group so you get a little more savings on your colder data.
Of course that’s going to give you a little less performance when you’re reading that cold data that’s already compressed. But it’s cold data, right? About 14 days is the threshold. So that’s gonna do in the larger compression group. Now that was 9.8, but at 9.10.1 we actually added a more aggressive compression algorithm.
So z standard compression, which is – I don’t know if everybody knows the Facebook z standard compression algorithm – but we’re using that to actually compress the cold data and get a lot more savings out of it.
Justin Parisi: Okay. So it sounds like that we’re looking at storage efficiency from a different perspective than we used to, where basically you’re taking the concept of is data active?
But what happens if we start to access that data later on? Like, is that gonna have some impact, or what is that gonna do?
Chris Hurley: Yeah, it is. Right. So you’re, especially because we’re using the larger compression groups, when you’re doing small reads, of that cold data, you’re gonna have to reap the whole compression group, right?
So there’s a little bit of a penalty there, but if you do it enough times, well there is a mechanism to uncompress it back to the smaller compression group. You still get some savings, but it allows for ONTAP to eliminate that penalty as much as possible.
Justin Parisi: So can you kind of tell me a little bit more about these compression groups? Like what are the main differences and is it something that I can modify later on? Can I make the compression group work differently? Or is it just something that’s baked into ONTAP?
Chris Hurley: It’s basically something that’s baked into ONTAP.
So by default inline, what we do is we take data and look at 8K chunks of it, right? You know, ONTAP writes in 4K blocks. So we look at 8K chunks of it and if we can compress that down, of course, into only one 4K block, we’re only gonna consume one 4K block in ONTAP.
Now in the TSSE model after 14 days, we’ll re-look at that data and then use 32K compression blocks and that way if we can compress eight 4K blocks down to seven 4K blocks, or six or five or even less, right? That’s even more savings. So, those larger compression groups from 8K inline to 32K cold TSSE compression groups, that’s where you get your savings.
And of course the Z standard compression algorithm on that cold data.
Justin Parisi: So what kind of storage savings can I get out of this? How much does it improve how much I save from previous ways we did it?
Chris Hurley: Well, it all depends, right? Every data set has its own compressibility, but by default, the Z standard algorithm has a lot more aggressiveness in its compression algorithm than what we normally use in ONTAP, which is lzopro, and that is also what we use for the inline 8K. You know, I hate to put down a number, but there is some significant increase in the data reduction with Z standard.
Justin Parisi: And what sort of workloads see better overall results?
Like I, I know that some workloads are gonna be more efficient than others. Where would you see more compression savings versus less compression savings.
Chris Hurley: So things with repetitive patterns, of course, like VDI, and VMware, and those type of workloads, would greatly increase their data reduction with the Z standard compression.
But, others as well, because the compression algorithms, as they mature, they are able to compress more and different types of data. So as the version increases of that compression algorithm, you’re gonna be able to compress more and more data.
Justin Parisi: So that sounds pretty cool. Are we looking at even further advancements with compression algorithms? Are we gonna add any new things in the future? I mean, I know you can’t commit to things, but is that something that we’re looking at?
Chris Hurley: Oh, absolutely. We’re always looking to improve on that.
Right, because what does that translate to? That translates to savings for our customers, right? If you wanna store a hundred terabytes of worth of logical data and you get a 3:1 compression ratio, then you only need, what, 33 terabytes of storage, and you can store a hundred terabytes of data.
That’s a benefit to our customers. Now, like I said, we’re always looking at something new, bringing that to the next level and finding new ways to reduce the data in ONTAP, so, A) you don’t have to buy as much storage to store the same amount of logical data, and B) maybe even get some performance increase for new technologies for the storage efficiencies.
Justin Parisi: Are we still doing that guaranteed storage efficiency deal where we basically say you’re gonna get 3:1 or 4:1 with certain workloads? Is that something that we’re still doing today?
Chris Hurley: Yeah, we are, we’ve recently also changed that to a blanket 4:1 guarantee for SAN workloads.
So anything that you run SAN protocol on, you know, your block storage will get a 4:1 guarantee by default when you entered the program. And for NAS it is 1.5:1 on that same program.
Justin Parisi: So why is there such a big discrepancy between SAN and NAS for those particular storage efficiency ratios?
Chris Hurley: NAS of course, everybody knows things called unstructured data, right? And that data compressibility and de-dupeability, it is a lot less than your structured data workloads, your Oracles, your MSSQL, your VMwares, and anything else that runs on the block protocol. So we have to account for those things.
Justin Parisi: All right. So it sounds like storage efficiency is pretty good at this point. We’re getting better with each release. So one feature that we actually have that kind of, I wouldn’t say it’s a storage efficiency feature, but it helps save space because you’re not creating additional data sets.
You’re basically just creating a sparse cache, and that’s gonna connect to an origin data set. So, you know, instead of creating an exact replica of say, a hundred terabytes, you’re creating maybe a cache that’s one gig or two gigs, or 10 gigs, whatever. Right. So that’s traditionally been for mostly reads, and that’s FlexCache.
Now we have a new feature in ONTAP 9.12.1 that allows us to do even more with the FlexCache architecture. So what is that, Chris?
Chris Hurley: So we’ve got what we’re calling write performance. It’s not quite write back. So FlexCache, as you mentioned, right? It’s a sparse read-writeable volume, but up until now when you write at a FlexCache, that write gets transferred all the way over the wire to the origin and has to be acknowledged by the origin before it can be acknowledged back to the client.
Well, with 9.12.1, we now have the ability to acknowledge that write and store it locally at the FlexCache volume and acknowledge that immediately to the client that’s writing to the FlexCache volume without having to wait for that traversal all the way to the origin.
Justin Parisi: So you say it’s not quite write back, but that’s what we’re calling it.
So why is it not "write back"? Why is it just "write performance"?
Chris Hurley: Well, the origin still has to be involved with inode creation to file creations, and it also still has to be involved with opens, you know, locking and that sort of thing. So the main operations that are actually accelerated are the actual write operations.
Justin Parisi: So with a FlexCache, you might have multiple caches pointing at a single origin, and I would imagine that applications that leverage the write back or the write performance aspect of it are gonna have multiple writes going to the same files in some cases. So how is that handled?
Chris Hurley: It’s handled very similarly to how it’s handled today, right?
So we have the write lock delegations that come to each cache and then only once that write lock delegation needs to be revoked by the origin, the cache also needs to flush the temporary data that has been written at the cache. So, you look at the write lock distribution, the way it sits now in FlexCache, it just means that the write lock stays a little longer at the cache, and so in order for it to be able to write at the cache and then flush it back to the origin, when that write lock is revoked.
And how is
Justin Parisi: it handled on the origin, if we’re changing the files there? Does the cache get evicted or is there a lock in place then as well?
Chris Hurley: Let’s say you’re writing at cache one, as soon as there’s a write request from either the origin or another, the origin, it’s gonna ask for that write lock back at the cache where that file is being written to. And at that point, once the cache does relinquish that write lock, that’s when the data gets flushed. So the next write can happen and that lock will go back to the origin or to the other cache.
So it keeps it all in order that way.
Justin Parisi: So let’s say there’s a weird rare event where three clients have three cachea and they’re all trying to write to the same file and at the same time, we’re writing a file in the origin. Who wins there?
The one who has the write lock. Right? Okay.
Chris Hurley: That’s the only way you can write is if you have the write lock to the file.
Now there is one of those weird scenarios where, as you’re writing, there’s some disconnection that happens, right? And at that point you have the option to go ahead with writes somewhere else. Because basically the file is held hostage by the disconnected cache. But then, what happens there is, you basically lose your writes.
You have to force that though. If you have a disaster and the cache cluster actually goes away and can’t come back, you can force that to happen.
Justin Parisi: So I would imagine that you would just go in and clear the locks from the vserver using like a vserver locks command?
Chris Hurley: It’s not quite a vserver locks, it’s a FlexCache command. But yeah, same concept.
Justin Parisi: Okay. And viewing those locks, would that also be a FlexCache command or
Chris Hurley: is that a vserver locks command?
That is a vserver locks command.
Okay.
Justin Parisi: So what happens if I run a vserver locks command to clear the lock? Is that gonna not allow it to clear on the FlexCache or does that override whatever’s on the FlexCache lock command?
Chris Hurley: Well normally the locks are flushed. Right. Normal lock operations, you can use vserver locks. But if you need to force a FlexCache to be able to move ahead, you have to use FlexCache commands.
Justin Parisi: Okay. And is there like a lock timeout period? Like a grace period on the FlexCache where those locks will eventually release if they can’t be reestablished?
Chris Hurley: Under normal operations without write back, there is a two minute timeout. But that’s not for the write back, it is a set it and forget it kind of thing, right? So the lock stays forever unless it needs revoked. But for normal write operations where the write around operations because there’s nothing pending at the cache, after two minutes, there’s a timeout and all the locks are revoked for that disconnected cache.
Justin Parisi: And is there a way to disable the idea of write back to the FlexCache? Like can you go back to the old behavior? Is there a way to tell it, to just do what it used to do, or is that just something that’s baked into the new release?
Chris Hurley: So by default it stays the same? Right? Write back is an option. And we only recommend write back for, for more than 10 milliseconds of latency between the FlexCache and the origin. Because of the complexity, it provides actually negative performance increase for anything that is less than about 10 milliseconds of latency between the origin and the cache.
Justin Parisi: Okay. So if you’ve got really a low latency between the origin and cache, you’re better off just using the old behavior because it’s just gonna add performance hits that you wouldn’t see normally.
Chris Hurley: Exactly.
Justin Parisi: All right. Makes sense. So as far as the new feature, from what I understand, it’s a public preview in ONTAP 9.12.1, or is that available all the way?
Chris Hurley: So currently it is a public preview. That just means that feature has gone through only limited testing. We’ve gone through most regressions, but there are still things that could happen, right? That’s why we’re calling it a public preview.
But we foresee that we can release it as a GA in a P release after 9.12.1.
Justin Parisi: Okay, cool. So 9.12.1 will eventually have available for everybody, but in the meantime, you probably don’t wanna put production workloads on this right now. It’s more of a kick the tires thing.
Chris Hurley: Correct.
Justin Parisi: Okay. So what sort of use cases are gonna get the most benefit out of this? Like what industries are asking for this type of thing the most?
Chris Hurley: Well, of course FlexCache in the EDA is hot. Right? And that is really one of those use cases that’s really hot and loves to use FlexCache, but it’s their cloud bursting, and it’s not only EDA, right? It’s anything that uses cloud bursting, but needs some writes at the cloud. So, you know, traditionally our cloud bursting has more around reading the data from OnPrem in the cloud than doing some work on it and then just writing back some little results and that’s okay.
But there’s a lot more use cases to where there are constant writes at the cache during the cloud bursting. So using the cloud as it’s supposed to be used, for capacity bursting or core bursting or however you want to do it. And that is where the write back actually works a lot better if you have to do performant writes as you’re bursting in the cloud.
You don’t just need to wait for some results and have that write back, write around all the way to the origin. You want some performant writes in that whole workflow while you’re bursting up in the cloud. That’s one of the biggest use cases we’ve seen around this.
Justin Parisi: Yeah, and I guess you’d be more likely to see that 10 millisecond latency in the cloud versus doing something in a data center.
Chris Hurley: Right, right. Yeah. The data center is usually five milliseconds or less, or not even five. And for those you don’t want to use FlexCache write back.
Justin Parisi: So that said, how would I know what my latency is? Is there an easy way to find out from ONTAP what the latency between the cache and the origin is, or is that something that’s hard to figure out?
Chris Hurley: No, it’s actually fairly easy. So in order to create a FlexCache, you have to peer the clusters anyway, right? And ONTAP does have a cluster peer ping command. So you can use that to determine what your latency is between the two clusters.
Justin Parisi: Okay, cool. So cluster peer ping will tell me roughly what my latency is gonna look like and what I should expect for something like FlexCache.
Chris Hurley: Yep.
Justin Parisi: And that goes for other things too as well, right? I mean, we’re looking at SnapMirror and that sort of thing. There’s Synchronous Snap Mirror, which also has a requirement of 10 milliseconds or less. So it’s not just FlexCache that has this limitation of 10 milliseconds. It’s pretty much across the board for any time you’re doing something remotely over a WAN.
Chris Hurley: Yeah, exactly.
Justin Parisi: And I would imagine this supports both SMB and NFS use cases, including NFSv4?
Chris Hurley: Absolutely. All the protocols.
Justin Parisi: Cool. And, and another thing to add, I know we added ONTAP S3 exposure to NAS volumes. So am I able to expose a FlexCache to object, or do I have to do that to the origin?
Chris Hurley: Well, as of right now, it’s only supported at the origin. We haven’t qualified it for use at the cache yet.
Justin Parisi: All right. And we had something called I guess mirroring of caches? Am I imagining that?
Chris Hurley: What you probably are talking about is the ability to have the same MSID and the same file handles between caches.
Is that what you’re talking about?
Justin Parisi: It might be. I seem to remember there was some sort of ability to have some sort of cooperation between caches or mirroring of caches. So tell me a little bit more about how we can do that.
Chris Hurley: So one of the things that you can do is, and we call it FlexCache DR but it’s kind of…
Justin Parisi: That’s what it was.
It was DR. And it throws you off. Yeah. That’s why I thought it was mirroring, so, yes.
Chris Hurley: Yeah, exactly. But it’s a way for clients to be able to seamlessly move from one cache to another if the cluster were to go down. So when you think of the FlexCache, all the writes are write around, so they end up at the origin and it’s a sparse volume.
It’s not a critical production thing, right? It is, of course. But in order to best insulate from failures or anything like that, you can create another FlexCache in the same data center or area, right? Logical location. And then what it could do is both caches will have the exact same file handles.
Of course, they have to be in different SVMs, right? If you do it in the same cluster, they have to be in different SVMs, but that way the NFS client, can seamlessly move from one cache to another without getting that pesky, stale file handle error.
Justin Parisi: So how does it handle the network aspect of that?
Is it automatic failover or do you have to remount?
Chris Hurley: No, you don’t have to remount. What you have to do is either in automounter or in DNS or somehow you need to be able to flip your clients to point from one SVM to the other, whether it’s in the same cluster or different clusters. That is up to the customer to configure that stuff.
Justin Parisi: So, I know when we mount, we have an IP address established, and that’s a TCP connection established. If there’s a cache failure, do we have to have the same IP in that SVM or can we use a different IP address? How does that fail over happen? It’s not quite clicking for me there, like how do we get that LIF fail over in addition to that?
Chris Hurley: Well you can bring the same IP up in different clusters if you want, or different SVMs. Elliot, you’ve done some work on that, right?
Elliott Ecton: Yeah, I have. So there’s a couple ways to do it. One way is if you have two different clusters you just keep one down and we’ll call it the DR cluster, right? And then when you have a failure to your main site, you would just Admin down that IP address and bring it up on the DR cluster and then traffic will start going to that. So it’s pretty seamless, and the NFS clients don’t require a remount and they just keep plugging along.
Justin Parisi: Okay. So basically you’re just using the same IP address and then we figure it out based on the network. Now I imagine that the MAC addresses are gonna be different. So, is there any ARP cache clearing that I have to do, or does that client just handle it automatically?
Elliott Ecton: Everything will be updated by gratuitous ARP when the IP goes up. So everything in the network will be updated where the new IP address lays.
Justin Parisi: Okay, cool. So the IP address comes up, says, "Hey, I’m here! Here’s my new MAC address! Use this instead." Right.
Elliott Ecton: You know, there’s more ways to do it than that. That’s the simplest way to configure it. But you can also do BGP VIPs and bring up a VIP over there. Just other things like that. There’s quite a few ways that you can migrate the IP. Load balancers are another way to see if there’s liveliness to the IPs. Several solutions.
Justin Parisi: Okay. So you mentioned BGP, and that’s something that doesn’t get a lot of press, right? It’s something that just kind of works. So how would that work in ONTAP? Is there anything special I need to configure for that to work? Or is it all on my network side?
Elliott Ecton: So there’s very little to do on the ONTAP side. BGP is a very deep topic, but ONTAP has a very simplistic implementation of it.
To bring up BGP, you need to peer with your routers and you have to work with your network team to get different ASNs and some IP addresses to establish the BGP peer session. And then you just create a VIPon top of that. And there’s no layer two, right?
So it doesn’t have to be in a subnet. This VIP just kind of floats where the BGP peer connections are. So it makes it really easy to migrate an IP address from one cluster to another. And when you do that, all your BGP routers, all your BGP peering, most of which your network team has set up, will update the BGP paths to get to the new cluster.
Justin Parisi: Okay. And it sounds like that might be a little more seamless than having to manually do things on your end.
Elliott Ecton: Yep, exactly.
Justin Parisi: So this FlexCache DR piece. Now I know it’s not mirroring necessarily, but with mirroring we have this concept of read only access to destination volumes. Is that something that carries over to FlexCache? Can we do write back to these caches that are doing DR?
Chris Hurley: Unfortunately with a FlexCache DR, that is a read only cache, right? Those aren’t available to write just because of the complexity with the MSIDs and those type of things. So yeah, for now it is a read only cache.
Justin Parisi: Okay. So just limited use case for now, but still it’s valuable, right? I mean, we were able to keep things going and running in case of outage.
Chris Hurley: Oh yeah, absolutely helps for tools distribution, binary distributions, things like that. I mean, it’s totally valuable for those.
Justin Parisi: So as the FlexCache write back goes, is there any other consideration I need to keep in mind, Chris?
Chris Hurley: Well, yeah. So there’s the amount of write back you have in your caches, right?
You want to keep that to a minimum. As I mentioned, by default it is not write – cuz there is some heaviness to it. Right. There is some heaviness to this feature. So you want to use it intelligently and where it’s really needed.
Justin Parisi: Okay. I know that with these early access previews, we let customers kick the tires on these, did you have any customers running this particular feature?
And if so, what was their feedback?
Chris Hurley: Not yet. Right? So actually FlexCache write back isn’t going to come in until GA, so 9.12.1 GA is when it actually drops in ONTAP. So as soon as that happens, we’re gonna get some customers on it and get their feedback.
We did get their feedback from the concept and we got all smiles and all thumbs up.
Justin Parisi: Okay. And you said it’s not in the RC release, it’s only in the GA release?
Chris Hurley: Correct.
Justin Parisi: Okay. So if I’m spinning up a new ONTAP release, I gotta wait for GA to, to have the access to this.
Chris Hurley: Mm-hmm. Unfortunately.
Justin Parisi: I would imagine that we have a technical report to talk about FlexCache. What TR number do we have for that?
Chris Hurley: That is TR 4743.
Elliott Ecton: It is getting updated soon. It’ll have the write performance part updated pretty soon by time GA rolls around.
Justin Parisi: Will we have any sort of performance numbers that we can look at to see what kind of performance we can expect?
Or is that just something that you have to kind of test out in your environment?
Elliott Ecton: There is some testing that’s gonna be going on that is actually gonna be starting sometime in the next week or two. So we should ask some performance numbers for you to look at.
Justin Parisi: All right, cool. And yeah, it’s just a baseline, right?
Like an idea of like, oh, generally this’ll work. Cause I know that every environment is different, but people do like to have some idea if it’s gonna be better or worse, if it’s worth my time. So, yeah. All right, so sounds like FlexCache write back or write performance, whatever you wanna call it, is gonna be pretty promising here.
And I imagine we’re gonna build onto things later on in future releases. Again, Chris, if we wanted to reach you, how do we do that?
Chris Hurley: I am online, Twitter and other places @AverageguyX and Christopher.Hurley@netapp.com.
Elliott Ecton: I am very not much on social media, so we’ll just do my NetApp email, which is Elliott@ netapp.com. Two Ls, two T’s in Elliott.
Justin Parisi: Cool. We’ll include that in the show notes so people don’t have to worry about spelling it and they can just click on the link and email you guys.
All right, that music tells me it’s time to go. If you’d like to get in touch with us, send us an email to podcast@netapp.com or send us a tweet @NetApp.
As always, if you’d like to subscribe, find us on iTunes, Spotify, Google Play, iHeartRadio, SoundCloud, Stitcher, or via a TechONTAPpodcast.com. If you liked the show today, leave us a review. On behalf of the entire Tech ONTAP podcast team, I’d like to thank Elliot Ecton and Chris Hurley for joining us today. As always, thanks for listening.
[podcast outro music]
Why Is The Internet Broken? 
Pingback: A Year in Review: 2022 Highlights | Why Is The Internet Broken?