Netapp : How to restore data from aggregate snapshot

          Today one of our user found himself in wet pants when he noticed his robocopy job has overwritten a folder, rather than appending new data to it. Being panicked he run to me looking for any tape or snapshot backup of his original data, which unfortunately wasn't there as previously he confirmed that they don't need any kind of protection.

Now at this time I had only place left where I can recover the data, aggregate level snapshots; so I looked at aggregate snapshots and saw it goes back to time when he had data in place. Knowing that the data deleted from volume is still locked in aggregate's snapshot I was feeling good that I have done a good job by having some space reserved for aggregate level snapshot, which no one ever advocated.

Now the next step is to recover the data, but problem was that if I revert aggregate using "snap restore –A" then all the volumes in that aggregate will be reverted which will be bigger problem. So had to go on a different way, use aggregate copy function to copy the aggregate's snapshot to an empty aggregate and then restore the data from there.

Here's the cookbook for this.

Pre-checks:

• The volume you lost data from is a flexible volume
• Identify an aggregate which is empty so it can be used for destination (could be on another controller also)
• Make sure the destination aggregate is either equal or larger than source aggregate
• /etc/hosts.equiv has entry for the filer you want to copy data to and /etc/hosts has its IP address added, in case of copying on same controller loopback address (127.0.0.1) should be added in /etc/hosts file and local filername should be in hosts.equiv file
• Name of aggregate's snapshot which you want to copy

Example:

Let's say the volume we lost data was 'vol1', the aggregate which has this volume is 'aggr_source', the aggregate's snapshot which has lost data is 'hourly.1' and empty aggregate where we will be storing data to is 'aggr_destination'

Execution:

• Restrict the destination aggregate using 'aggr restrict aggr_destination'
• Start the aggregate data copy using 'aggr copy start –s hourly.1 aggr_source aggr_destination'
• Once the copy is completed online the aggregate using 'aggr online aggr_destination'
• If you have done copy on same controller, system will rename the volume 'vol1' of 'aggr_destination' to 'vol1(1)'
• Now export the volume or lun and you have your all lost data available.

So here's the answer to another popular question, why do I need to reserve space for aggregate level snapshot. Do you have the answer now?

NetApp command line shortcuts

Just a few commands which I use frequently while on console.

 

CTRL+W = It deletes the word before cursor

CTRL+R = Rewrites the entire line you have entered

CTRL+U = Deletes the whole line

CTRL+A = Go to start of the line

CTRL+E = Go to end of the line

CTRL+K = Delete all the following texts

A few more commands are there but I feel arrow keys work better then you press these sequences like

CTRL+F = Right arrow

CTRL+B = Left arrow

CTRL+P = Up arrow

CTRL+N = Down arrow

CTRL+I = Tab key

Am I missing anything else?

FOR MORE 

How to copy files in Ontap

 As soon as someone asks this question we all say 'use ndmpcoyp' but what if you don't have any network adapters configured, will ndmpcoyp work? 
   
    No; ndmpcopy is very useful if you want to copy a file or a whole volume however one thing very few people know that it doesn't work if you don't have loopback adapter configured because ndmpcopy passes all the data through lo adapter so it's not only dependent on lo's availability, its speed also. So how do you copy the data if lo is not available?

The answer is simple, use dd, just an old fashioned unix command which does lot of thing, not only it can copy the file with full pathname you can even use block number and disk number and the best part, syntax is simple 'if' for from and 'of' for to.

It can be used not only for copying file around the system, in fact you can use it for testing I/O and copying file from snapshot also and this command can be used regardless of permission.

A little note, if you are afraid of going in advanced or diagnostic mode better keep use rdfile and wrfile because this command is not available in admin mode so you have to go in advanced mode to use this.

Here's the syntax of this command.

dd [ [if= file ] | [ din= disknum bin= blocknum ] ] [ [of= file ] | [ dout= disknum bout= blocknum ] ] count= number_of_blocks

Another note, if you are using count make sure you are using in multiply of 4 because a WAFL block size is 4k.

Example:

sim1> priv set advanced
sim1*> dd if=/vol/vol0/.snapshot/hourly.2/etc/snapmirror.conf of=/vol/vol0/etc/snapmirror.conf1

NetApp Active/Active vs. Active/Passive (Stretch MetroCluster) solution

Active / Active Controller Configuration

         In this configuration both the systems are connected to each other's disk and having heartbeat connection through NVRAM card. In the situation of one controller failure other controller takes over the loads of failed controller and keeps the operation going as it's having connection with failed controller's disk shelves.

Further details of Active / Active cluster best practices can be found in TR-3450

Active / Passive (Stretch MetroCluster) Configuration

        This is the diagram of active/active metrocluster, however the same design applies to active/passive metrocluster also except one node on the cluster is having only mirror of primary system's data.

       In this configuration primary and secondary systems can extend upto 500m (upto 100km with Fabric MetroCluster) and all the primary system data is mirrored to secondary system with Sync Mirror, in the event of primary system failure all the connection automatically gets switch over to remote copy. This provides additional level of failure protection like whole disk shelf failure or multiple failures at same time, however this needs another copy of same data and exact same hardware configuration to be available for secondary node.

      Please note that cluster interconnect (CI) on NVRAM card is required for cluster configuration however 3170 offer a new architecture that incorporates a dual-controller design with the cluster interconnect on the backplane. For this reason, the FCVI card that is normally used for CI in a Fabric MetroCluster configuration must also be used for a 31xx Stretch configuration.

Further details of MetroCluster design and implementation can be found in TR-3548

Minimizing downtime with cluster

      Although having a cluster configuration saves from any unwanted downtime however a small disruption can be sensed on the network while takeover /giveback is happening which is approximately less than 90 seconds in most of the environments and it keeps the NAS network alive with few "not responding" errors on clients.

A few points in related with this are given below:

CIFS: leads to a loss of session to the clients, and possible loss of data. However clients will reconnect the session by themselves if system comes up before the timeout window.

NFS hard mounts: clients will continue to attempt reconnection indefinitely, therefore controller reboot does not affect clients unless the application issuing the request times out waiting for NFS responses. Consequently, it may be appropriate to compensate by extending the application timeout window.

NFS soft mounts: client processes continue reconnection attempts until the timeout limit is reached. While soft mounts may reduce the possibility of client instability during failover, they expose applications to the potential for silent data corruption, so are only advised in cases where client responsiveness is more important than data integrity. If TCP soft mounts are not possible, reduce the risk of UDP soft mounts by specifying long retransmission timeout values and a relatively large number of retries in the mount options (i.e., timeo=30, retrans=10).

FTP, NDMP, HTTP, backups, restores: state is lost and the operation must be retried by the client.

Applications (for example, Oracle®, Exchange): application-specific. Generally, if timeout-based, application parameters can be tuned to increase timeout intervals to exceed Data ONTAP reboot time as a means of avoiding application disruption.