Introducing: Copy-Free Transition

Clustered Data ONTAP 8.3.2RC1 was announced last week and included many enhancements to ONTAP, including a feature called Copy-Free Transition.

A number of people knew about this feature prior to the cDOT release because they attended Insight 2015 and witnessed either a live demo of the feature or the session presented by Jay White (CR-2845-2: Using the 7-Mode Transition Tool).

We talked a bit about CFT in Episode 15 of the Tech ONTAP Podcast.

There’s also a video demo of Copy-Free Transition available here:

If you’re not familiar with Copy-Free Transition (CFT), then here’s a brief rundown…

cft-button

What is Copy-Free Transition?

Prior to cDOT 8.3.2, transition to clustered Data ONTAP involved copying your data from 7-Mode to clustered Data ONTAP using one of the many tools available. Architecturally and structurally clustered Data ONTAP is very different from 7-Mode which precluded the ability to upgrade in-place to clustered Data ONTAP.

Essentially, you would use one of the following migration options:

  • Use the 7-Mode Transition Tool (7MTT) which leverages SnapMirror to replicate data from 7-Mode to clustered Data ONTAP
  • An application-based migration option (such as Storage vMotion from VMware)
  • File copy options such as ndmpcopy, RoboCopy, rsync, etc.
  • Using Foreign LUN Import

As the above migration options are all methods that copy data, the general term used to describe them is Copy-Based Transition (CBT).

With CFT in 8.3.2 and later, the 7MTT can be used to migrate to clustered Data ONTAP by simply halting your 7-Mode systems, recabling your disk shelves to a cDOT system, then importing the data and configuration into the cluster.

Voilà! Transition simplified!

Why do we want to use CFT?

For starters, you’d use CFT because it allows you to move a large amount of data in a fraction of the time it would take you to copy it. This “big bang” type of transition does require a little extra planning to make sure the clustered Data ONTAP environment is functional post-CFT, but the 7MTT contains extensive pre-checks and assessment capabilities to assist you with your transition planning.

Our live demo at Insight involved a 2-node HA pair with 2 data aggregates and 4 volumes. These volumes served NFS, CIFS and iSCSI data. We were able to finish a live migration in less than 30 minutes, start to finish.

I wasn’t just wearing a Flash costume for giggles – I wanted to emphasize how fast CFT can be.

cft-flash

The guidance from engineering I’ve heard is 3-8 hours, but they’ve been *very* generous in the amount of time built in for cabling the shelves. The time to completion is also dictated by the overall number of objects in the system (ie, number of volumes, qtrees, quotas, exports, etc) and not the size of the dataset. That’s because the 7MTT has to build the configuration on the cDOT system and that takes a number of ZAPI calls. Fundamentally, the message here is that you can do CFT, and roll back if necessary, within a single maintenance window. The main contention for timing here will be how long it takes to re-cable or move disk shelves and reconnect clients.

The actual conversion of the 7-Mode volumes is relatively quick.

Anecdotally, I heard about a customer that did an early preview of CFT with multiple terabytes of data. The cutover after the shelves were moved took 30 minutes. That is… impressive.

That timing is not guaranteed, however – it’s a good idea to plan the 3-8 hours into your window.

Aside from the time it takes to transition, using CFT is also a bonus for people who did not want to purchase/rent swing gear to move data (aside from the minimal amount of equipment needed to bring the cDOT cluster up), or people that simply wanted to keep their existing shelves that they already had support on.

Rather than having to copy the data from 7-Mode to a swing system and then to a cDOT system, you can now simply use the existing gear you have.

The sweet spot for CFT is really unstructured NAS data, such as home directories. These datasets can potentially have thousands or millions of objects with corresponding ACLs. CFT allows for a massively simplified transition of this type of data.

 

What do I need for CFT?

This is a short list of what you currently need for CFT. Keep in mind that the product documentation for the cDOT release is the final word, so always check there.

Currently, you need:

  • 7-Mode 8.1.4P48.1.4P9 (source system)
  • Clustered Data ONTAP 8.3.2RC1 or later (destination)
  • 7MTT 2.2 or later
  • 64-bit aggregates
  • A minimally pre-configured* storage virtual machine on the destination cluster – one per vFiler/node
  • If using CIFS, a CIFS server on the destination
  • An HA pair with no data on it other than the cluster config/SVM placeholders
  • Functioning SP modules on the 7-Mode systems

*Minimally pre-configured here means you need a vsroot volume. If CIFS is involved, you need a data LIF, DNS configuration and a CIFS server pre-created in the same domain as the source CIFS server.

If you have a cluster with existing data on it, you can still use CFT, but you have to have a 4 node cluster with 2 of the HA nodes evacuated of all data. Otherwise, 7MTT won’t allow the CFT to continue.

For platform support, please check the documentation, as those are subject to change.

Also keep in mind that this is a version 1.0 of the feature, so there will be more support for things as the feature matures.

What isn’t currently supported by CFT?

  • SnapMirror sources and destinations are supported, but SnapVault currently is not.
  • MetroCluster is currently not supported.
  • 32-bit aggregates are not supported, but can be upgraded to 64-bit prior to running CFT.
  • Systems containing traditional volumes (TradVols), but let’s be real – who uses those still? 🙂
  • Currently, clusters with existing datasets are not supported (must have an evacuated HA pair)

What happens during the CFT process?

In our demo, we had the following graphic:

cft-process

In that graphic, we have gear images for automated processes and M for manual processes. The good thing about CFT is that it’s super easy because it’s mostly automated. The 7MTT handles most of it for you – even the halting of the 7-Mode systems.

Here’s a rundown of each part of that flowchart. For more details, check the product documentation and TR-4052. (not updated yet, but should be updated in time for 8.3.2GA)

Keep in mind that during the 7MTT run, each section will have a window that shows exactly what is happening at each phase.

Start CFT Migration

This covers the start of the 7MTT and the addition of the 7Mode HA pair and cluster management LIF to the tool. This does not cover the initial up-front planning prior to the migration, so keep that in mind. That all has to take place before this part.

During the “Start CFT” portion, you will also populate the data LIFs you want to migrate, the volumes and define the volume paths. You will also map the vFilers you are migrating to the SVM placeholders on the cluster.

Planning and Pre-checks

This portion of CFT is an automated task that will look at a list of pre-canned checks of 7-Mode and cDOT to ensure the source and destination are ready. It checks compatibility via a series of pre-canned checks and looks to see if 7-Mode is doing things that are not currently supported in cDOT. If anything fails, the tool makes you correct the mistakes before you continue as not to allow you to shoot yourself in the foot.

Apply SVM Configuration

This automated process will take the information grabbed from 7-Mode and apply it to cDOT. This includes the data LIFS – they will get created on the SVM and then placed into a “down” state to avoid IP conflicts.

Test SVM Configuration

Here, you would manually ensure that the SVM configuration has been applied correctly. Check the data LIFs, etc.

Verify Cutover Readiness

This is another pre-check that is essentially in place in case you did the pre-check a week ago and need to verify nothing has changed since then.

Disconnect clients

This is a manual process and the start of the “downtime” portion of CFT – we don’t want clients attached to the 7-Mode system during the export/halt phase.

Export & Halt 7-Mode Systems

This is an automated process that is done by the 7MTT. It leverages the SP interfaces on the 7-Mode systems to do a series of halts and reboots, as well as booting into maintenance mode to remove disk ownership. We’re almost there!

Cable Disk Shelves

Another manual process – you essentially move the cables from the 7-Mode system to the cDOT system. You might even have to physically move shelves or heads, depending on  the datacenter layout.

Verify Cabling

This is an automated 7MTT task. It simply looks for the disks and ensures they can be seen. However, it’s a good idea to do some visual checks, as well as potentially make use of Config Advisor or the 7MTT Cabling Guide.

Import Data & Configuration

This automated phase will assign the disks to the cDOT systems, as well as import the remaining configuration that could not be added previously (we need volumes to attach to quotas, etc… volumes had to come over with the shelves). This is also where the actual conversion of the volumes from 7-Mode style to cDOT style takes place.

Pre-prod verification

This is where you need to check the cDOT cluster to ensure your transitioned data is in place and able to be accessed as expected.

Reconnect clients

This is the “all clear” signal to your clients to start using the cluster. Keep in mind that if you are intending on rolling back to 7-Mode at any time, the data written to the cluster from here could potentially be lost, as the roll back entails reverting to an aggregate level snapshot.

Commit

This is the point of difficult return – once you do this, the aggregate level snapshots you could use to roll back will be deleted. That means, if you plan on going back to 7-Mode, you will be using a copy-based method. Be sure to make your decision quickly!

Rolling back to 7-Mode

If, for some strange reason, you have to roll back to 7-Mode, be sure you decide on it prior to committing CFT. In our demo, roll back was simple, but not automated by the 7MTT. To make the process easy and repeatable, I actually scripted it out using a simple shell script. Worked pretty well every time, provided people followed the directions. 🙂

But, it is possible, and if you don’t commit, it’s pretty fast.

If you have any questions about CFT that I didn’t cover here, feel free to comment.

Also, check out this excellent summary blog on transition by Dimitris Krekoukias (@dkrek):

http://recoverymonkey.org/2016/02/05/7-mode-to-clustered-ontap-transition/

Advertisements

Clustered Data ONTAP 8.3.2 is here! (plus, Insight EMEA 2015 recap)

It’s finally here!

Clustered Data ONTAP 8.3.GA is up and available here:

http://support.netapp.com/NOW/download/software/ontap/8.3.2

I can also finally talk about the live demo I was working on for NetApp Insight. I even made some videos for it:

 

As for what’s in the release, I’ll keep it high level here and start with what I was showing off…

Copy-Free Transition!

One of the biggest pieces of FUD being thrown around regarding NetApp was the fact that the transition from Data ONTAP operating in 7-Mode to clustered Data ONTAP was a “forklift upgrade.”

Given the fact that the operating systems were vastly different, it wasn’t 100% inaccurate, but it was definitely an exaggeration in my opinion.

However, one of the main things people kept asking for was a way to simply move the shelves from the old 7-Mode heads to the new cDOT heads.

“Shouldn’t it be easy?” they (rightfully) asked?

Well, now it is. The live demo we did showed a Copy-Free Transition (CFT) in real-time from 7-Mode to cDOT and it took less than 20 minutes.

Some other notable features of this release:

  • SAN Break-Resync-Break for 7àC Snapmirror relationships
  • MCC Improvements – Cisco 9250i switch support, sharing 8G card FC-VI/FC-Initiator
  • In-line Deduplication
  • Volume Re-host
  • QoS node limit increase (24 nodes!)
  • SVM DR MSID replication (filehandles!)

I wrote up a more in-depth post on CFT after Insight. Also, be sure to check out the TechONTAP Podcast episode where we cover all the new goodness of the 8.3.2 cDOT release!

Episode 15 – Clustered Data ONTAP 8.3.2RC1

NetApp Insight EMEA Recap

flash-netapp

NetApp Insight EMEA is all over and if you noticed I didn’t have any recaps posted, it was because I was INSANELY busy.

In addition to my sessions and the CFT demo, I also joined the TechONTAP podcast as a co-host, which is awesome – but when you have daily recaps, that means you are doing daily edits. So, not a lot of sleep involved.

If you’re interested, here are the daily recaps of the conference:

Day 1 Recap

Day 2 Recap

Day 3 Recap

Day 4 Recap

The Flash did make another appearance, and he even did a video with our All Flash FAS TME, Dan Isaacs called “Flash on Flash”:

I also had the opportunity to talk Data Fabric with NetApp A-Team member Paul Stringfellow:

Stay tuned for more information about 8.3.2 on this blog and by following me on Twitter @NFSDudeAbides, as well as listening to the TechONTAP podcast.

TECH::NetApp Tech ONTAP Podcast Announcement!

If you’ve noticed, I’ve done some work in recent months with the guys at the Tech ONTAP podcast. I’ve blogged about it some, done some graphics/photography work, and even appeared on a few episodes – most notably the NFS one.

My interest in the podcast started back when Nick Howell (@datacenterdude) originally ran it and I was just starting up my own blog/social media presence. Nick was kind enough to mentor me as I started out.

I considered trying my hand at doing my own podcast, but why re-invent the wheel? The Tech ONTAP podcast (formerly the NetApp Community Podcast) was doing great things already.

The work I did on the podcast was mostly out of interest in doing something a little different from my day to day (and I learned a bunch from it), but secretly I hoped that I could be a more regular part of it.

Well, my wish has come true – I am honored to announce I am now an official co-host of the NetApp Tech ONTAP podcast!

How it happened

In the process of my work with the podcast guys – Pete Flecha (@vPedroArrow), Glenn Sizemore (@glnsize) and Andrew Sullivan (@andrew_NTAP) – it turns out that I actually liked working with those guys. In my interactions, we all seemed to be on the same page with a lot of things, and they’re some of the best and brightest NetApp has to offer.

Because those guys are all so top-notch, I didn’t see a way in to be a more official part of the podcast – it was just too crowded!

So I decided I’d just keep helping out – because it was fun. But also because I wanted to show the guys what I could bring to the table.

However, a spot recently opened up – Pete Flecha decided to move on to a new position at VMware. And the podcast guys decided I’d be a good fit to try to replace him.

Now, that remains to be seen… but I plan on doing my best! I do have a background in doing radio (college, at least, at WKNC 88.1), so I don’t think the transition will be super difficult.

I’ll be helping with the editing, blogging, promoting and whatever else I can do to make it successful. I’ll also be trying to fill the role of Pete Flecha, which was conflict mediator between Glenn and Andrew. Those are some big shoes to fill!

If you happen to listen, feel free to send me feedback on what I can do better, tips on how to improve the podcast or just to tell me how I’m doing!

Check out the episode where we talk about SnapMirror to Everything and the Data Fabric, and where I get announced as joining the team!

Episode 14: Protecting your Data with SnapMirror to Everything

If you’re going to be at Insight Berlin, come find me at one of my sessions or doing some Tech ONTAP podcast work!

Email: podcast@netapp.com

TECH::Data ONTAP 8.2.4P3 is available!

Data ONTAP 8.2.4P3 has been released!

As the NFS TME at NetApp, I am pretty stoked.

Get it here:

http://mysupport.netapp.com/NOW/download/software/ontap/8.2.4P3/

The link above gives a list of bugs that are fixed in this P release.

For the product docs:

This is an update to both 7-Mode and clustered Data ONTAP. Keep in mind that minor releases (such as 8.2.x) are immediately considered GA, so there is no RC for minor releases. They are essentially patch rollups with minimal new features.

The update contains feature enhancements including:

  • Support for In-memory User to Group mapping database
  • GbE Support on UTA2 ports for data connectivity on Entry systems
  • RC4 Kerberos Encryption support in 7-mode
  • Scalability and Performance improvements for Parallel NFS mounts in LDAP environments
  • Bug fixes
  • Customer-0 with over 25k hours of run time

If you are using NAS, it’s HIGHLY recommended to upgrade to release, as there are numerous improvements to the NAS functionality.

If you are on a 8.1.x or prior release, consider beginning to plan on upgrading, as it will be a little more involved than a minor upgrade from 8.2.x.

TECH:: Lost in Space – Raw vs. Usable disk space allocation considerations

This is part 2 in the Lost in Space series on how space allocation works in Data ONTAP. This post covers the physical space aspect.

For the first post in the series, click here.

Raw vs Usable – Physical

In this post, we’re deciphering how much space you bought versus how much space you actually have available to use. The focus here is on physical space, before we even get to the software portion of ONTAP.

When you buy a disk, you pay a dollar amount per space unit. These days, it’s $ per GB. In the future, we’ll probably pay $ per PB. Slower drives with higher capacity (such as SATA) will cost less per GB. Faster drives with lower capacity (such as SSD) will cost more per GB. The good thing about technology, however, is that prices normalize over time as production of new stuff gets cheaper.

The file system

What most people don’t think about is that when you buy a hard drive, you are paying for what is considered raw space. The hard drive is useless until it is formatted with a file system. This is true of *any* storage device, whether it’s a Windows desktop using NTFS or an All-Flash FAS.

NetApp’s WAFL file system takes up ~10% of the raw drive space for its file system. NTFS takes up a minimum of 12.5%. Other storage vendors use varying amount of raw space for the file system, features, etc. The 10% is not modifiable.

RAID!

Hard drives in enterprise storage (or even servers) will also often participate in some sort of virtualized aggregation of disks to allow for better performance, redundancy, etc, generally known as RAID. In Data ONTAP, a default of RAID-DP (a double-parity variation of RAID6) is used, with the option of using RAID4.

What that means in terms of disk is this – Every time you create a group of disks for use with your storage system in ONTAP (known as a RAID group), you will lose at most, 2 disks for parity. That eats into your “usable” calculations.

RAID groups are contained within aggregates. An aggregate can have multiple RAID groups. These RAID groups can vary in size. The size of the RAID group will depend on what types of disks you have, what the intended workloads are, etc.

How much usable physical space do I really have?

raidgroup

Your usable disk space is considered to be:

(Number of disks – number of parity disks) x HDD raw space = Raw space adjusted for RAID

Then:

Raw space adjusted for RAID – (Raw space adjusted for RAID * 10%) = Total usable

For example….

If you have 2 RAID groups with 10 disks each and use RAID-DP, you essentially have 16 total disks for actual data use. 4 disks (2 for each RAID group) are dedicated to parity disks.

If those disks are 1TB each, that means you have 16TB of raw physical space.

Once you format the disks for use with WAFL, you then have 16TB – (16TB * 10%) = 14.4TB of usable physical space.

What’s important about physical space?

The key thing to remember about physical space is that no matter how much space your OS tells you you have available in a volume, share, LUN, etc. you are always going to be limited to the physical amount of space available in your system.

Future posts will cover the virtual aspect of storage space allocation. 

That about covers the physical space aspect of space allocation in Data ONTAP. If you have questions or want me to add something to this post, feel free to comment!

Stay tuned for more about space allocation in this series!

TECH::Lost in space – Space allocation in Data ONTAP

This is the first in a series of posts about how space allocation works in Data ONTAP. Because the topic is lengthy, I split the concepts up into multiple, easier to digest posts.

lostinspace

Space allocation in NetApp’s Data ONTAP is a question that comes up fairly often because it’s a concept that operates in three dimensions.

Physical – How many actual disks do we have?

Virtual – How much space have we told the software to allocate via FlexVols?

Actual – How much space are we actually using?

Then when you throw in things like snapshots, space guarantees, LUNs… it starts to get a bit muddled.

But before we get into how all that works, let’s talk about why it matters.

Why do I care about space allocation?

If you’re not monitoring your space in some capacity (pun intended), you’re doing it wrong. For Data ONTAP, NetApp already provides some space reporting utilities in the OnCommand, particularly rich with OnCommand Insight.

As a storage administrator, you have to ensure you know how much space you are using. If you run out of space, you can see issues with performance, write failures, backup or snapshot failures, or LUNs going offline.

Basically, if you ignore how much space you have, you are creating a potential outage scenario on your production storage. If you ignored the space problem for an indefinite amount of time, you may also have painted yourself into a corner because you didn’t have enough space to take backups.

In Data ONTAP, if you let your capacity approach 85-95%, you may start to see performance issues depending on your workload. Because WAFL is a filesystem that can write to any free block in storage, as the file system fills, it can take longer to find a free block to write to in some instances.

Space issues are not a concept that is unique to Data ONTAP – every storage vendor has a very similar set of storage capacity considerations that need to be monitored and addressed.

Lost in Space Table of Contents

Part 2: Raw vs. Usable disk space allocation considerations

This post covers how much space you actually start out with when you buy disks.

Stay tuned for more posts including:

Part 3: Snapshots and snapshot reservations

Part 4: Physical space allocation vs. Virtual space allocation

Part 5: Flexible Volumes, Space Guarantees and Space Allocation for NAS and SAN

Part 6: Space management in Data ONTAP

NetAppInsight:: #NetAppInsight 2015 – Berlin is just days away!

With NetApp Insight 2015 – Las Vegas in the books, my focus has shifted to the next big thing – NetApp Insight 2015 in Berlin!

In many ways, I enjoy the Insight shows in EMEA more than the ones in Las Vegas. Here’s why:

  • I don’t really like Las Vegas – at all.
  • I love Europe in general.
  • Tiergartenquelle
  • Warm pretzels
  • The stuff I work on – NFS and name services – are *huge* in Europe.

I consider what I work on to be the Hasselhoff of the protocols.

However, I got a lot of positive feedback on the stuff I did in Las Vegas. My sessions were generally well received and decently attended. The live demo we worked on had the most booth traffic of any booth at Insight.

And I even was able to meet the Flash…

For more information about how Vegas went, check this out:

https://whyistheinternetbroken.wordpress.com/2015/10/15/insight-vegas-home-stretch/

What I’ll be up to at NetApp Insight 2015 – Berlin

I have a feeling that as busy as I was in Las Vegas, I’ll be even busier in Berlin.

First and foremost… sessions!

This is my session schedule. Be sure to reserve your spot today with the NetApp Insight Schedule Builder if you want to see one of them.

1884-2: Unlocking the Mysteries of Multiprotocol NAS 

This is a level 2 session where I will attempt to demystify multiprotocol NAS and discuss some best practices with regards to clustered Data ONTAP. The feedback for this session was mostly good, but I’ve taken some of the constructive feedback and improved the session even more for Berlin. Come check it out!

  • Tuesday, Nov 17, 11:30 AM – 12:30 PM
  • Wednesday, Nov 18, 10:30 AM – 11:30 AM
  • Thursday, Nov 19, 2:15 PM – 3:15 PM

1881-3-TT: SecD Deep Dive

This is a level 3 session where I go pretty deep into how SecD works and how to use it to troubleshoot. Feedback from this session was overwhelmingly positive, as people felt this session was one of the most technical ones they had attended.

  • Monday, Nov 16, 2:45 PM – 3:45 PM
  • Wednesday, Nov 18, 2:15 PM – 3:15 PM

I’ve also got a Hands On Lab!

1804-2: Hands-On Lab for Configuring and Using Multiprotocol with LDAP in Clustered Data ONTAP® 

Live Demo

We’re also doing our live demo at booth C8 in the exhibitor hall. Be sure to come check out one of the hottest booths at NetApp Insight!

Our booth schedule:

insight-emea-demo-sched

Podcast!

tech-ontap-insight-berlin-sprechen

I’ll be helping the Tech ONTAP Podcast guys out a bit more in Berlin, so be sure to stop by the podcast booth to meet the podcast team, get a free podcast shirt and sticker!

Meet the Engineer

Schedule a meeting with me to chat about NFS, Name Services, clustered Data ONTAP or life in general!

Geek and Greet

Have you been NetApp certified? If so, you likely have an invitation to the NetApp Geek and Greet. I’ll be there Tuesday November 17, 5:30 – 6:30 p.m.

NetApp A-Team

In addition to all of the other stuff, I’ll be hanging out with the NetApp A-Team and helping out as much as I can. Come find me if you want more information on what we do and what it takes to be a member. Follow them on Twitter @NetAppATeam.

Finally, if you need to contact me or just want to see what I’m up to, follow me on Twitter @NFSDudeAbides.

For some highlights of last year’s Berlin trip, here are some photos:

Berlin 2014

See you all in Berlin!