Netapp : How to restore data from aggregate snapshot

          Today one of our user found himself in wet pants when he noticed his robocopy job has overwritten a folder, rather than appending new data to it. Being panicked he run to me looking for any tape or snapshot backup of his original data, which unfortunately wasn't there as previously he confirmed that they don't need any kind of protection.

Now at this time I had only place left where I can recover the data, aggregate level snapshots; so I looked at aggregate snapshots and saw it goes back to time when he had data in place. Knowing that the data deleted from volume is still locked in aggregate's snapshot I was feeling good that I have done a good job by having some space reserved for aggregate level snapshot, which no one ever advocated.

Now the next step is to recover the data, but problem was that if I revert aggregate using "snap restore –A" then all the volumes in that aggregate will be reverted which will be bigger problem. So had to go on a different way, use aggregate copy function to copy the aggregate's snapshot to an empty aggregate and then restore the data from there.

Here's the cookbook for this.

Pre-checks:

• The volume you lost data from is a flexible volume
• Identify an aggregate which is empty so it can be used for destination (could be on another controller also)
• Make sure the destination aggregate is either equal or larger than source aggregate
• /etc/hosts.equiv has entry for the filer you want to copy data to and /etc/hosts has its IP address added, in case of copying on same controller loopback address (127.0.0.1) should be added in /etc/hosts file and local filername should be in hosts.equiv file
• Name of aggregate's snapshot which you want to copy

Example:

Let's say the volume we lost data was 'vol1', the aggregate which has this volume is 'aggr_source', the aggregate's snapshot which has lost data is 'hourly.1' and empty aggregate where we will be storing data to is 'aggr_destination'

Execution:

• Restrict the destination aggregate using 'aggr restrict aggr_destination'
• Start the aggregate data copy using 'aggr copy start –s hourly.1 aggr_source aggr_destination'
• Once the copy is completed online the aggregate using 'aggr online aggr_destination'
• If you have done copy on same controller, system will rename the volume 'vol1' of 'aggr_destination' to 'vol1(1)'
• Now export the volume or lun and you have your all lost data available.

So here's the answer to another popular question, why do I need to reserve space for aggregate level snapshot. Do you have the answer now?

NAS on SAN ( Network Data Management Protocal )

       NDMP (Network Data Management Protocol) is an open protocol used to control data backup and recovery communications between primary and secondary storage in a heterogeneous network environment. 

       NDMP specifies a common architecture for the backup of network file servers and enables the creation of a common agent that a centralized program can use to back up data on file servers running on different platforms. By separating the data path from the control path, NDMP minimizes demands on network resources and enables localized backups and disaster recovery. With NDMP, heterogeneous network file servers can communicate directly to a network-attached tape device for backup or recovery operations. Without NDMP, administrators must remotely mount the network-attached storage (NAS) volumes on their server and back up or restore the files to directly attached tape backup and tape library devices.

   NDMP addresses a problem caused by the particular nature of network-attached storage devices. These devices are not connected to networks through a central server, so they must have their own operating systems. Because NAS devices are dedicated file servers, they aren't intended to host applications such as backup software agents and clients. Consequently, administrators have to mount every NAS volume by either the Network File System (NFS) or Common Internet File System (CIFS) from a network server that does host a backup software agent. However, this cumbersome method causes an increase in network traffic and a resulting degradation of performance. NDMP uses a common data format that is written to and read from the drivers for the various devices.

    Network Data Management Protocol was  originally developed by NetApp Inc., but the list of data backup software and hardware vendors that support the protocol has grown significantly. Currently, the Storage Networking Industry Association (SNIA) oversees the development of the protocol.

Common Internet File System (CIFS)

    Common Internet File System (CIFS) is a protocol that lets programs make requests for files and services on remote computers on the Internet. CIFS uses the client/server  programming model. A client program makes a request of a server program (usually in another computer) for access to a file or to pass a message to a program that runs in the server computer. The server takes the requested action and returns a response.

    CIFS is a public or open variation of the Server Message Block Protocol developed and used by Microsoft. Like the SMB protocol, CIFS runs at a higher level than and uses the Internet's TCP/IP protocol. CIFS is viewed as a complement to the existing Internet application protocols such as the File Transfer Protocol (FTP) and the Hypertext Transfer Protocol(HTTP). 

CIFS lets you:

• Get access to files that are local to the server and read and write to them
• Share files with other clients using special locks
• Restore connections automatically in case of network failure
• Use Unicode file names

NetApp command line shortcuts

Just a few commands which I use frequently while on console.

 

CTRL+W = It deletes the word before cursor

CTRL+R = Rewrites the entire line you have entered

CTRL+U = Deletes the whole line

CTRL+A = Go to start of the line

CTRL+E = Go to end of the line

CTRL+K = Delete all the following texts

A few more commands are there but I feel arrow keys work better then you press these sequences like

CTRL+F = Right arrow

CTRL+B = Left arrow

CTRL+P = Up arrow

CTRL+N = Down arrow

CTRL+I = Tab key

Am I missing anything else?

FOR MORE 

How to copy files in Ontap

 As soon as someone asks this question we all say 'use ndmpcoyp' but what if you don't have any network adapters configured, will ndmpcoyp work? 
   
    No; ndmpcopy is very useful if you want to copy a file or a whole volume however one thing very few people know that it doesn't work if you don't have loopback adapter configured because ndmpcopy passes all the data through lo adapter so it's not only dependent on lo's availability, its speed also. So how do you copy the data if lo is not available?

The answer is simple, use dd, just an old fashioned unix command which does lot of thing, not only it can copy the file with full pathname you can even use block number and disk number and the best part, syntax is simple 'if' for from and 'of' for to.

It can be used not only for copying file around the system, in fact you can use it for testing I/O and copying file from snapshot also and this command can be used regardless of permission.

A little note, if you are afraid of going in advanced or diagnostic mode better keep use rdfile and wrfile because this command is not available in admin mode so you have to go in advanced mode to use this.

Here's the syntax of this command.

dd [ [if= file ] | [ din= disknum bin= blocknum ] ] [ [of= file ] | [ dout= disknum bout= blocknum ] ] count= number_of_blocks

Another note, if you are using count make sure you are using in multiply of 4 because a WAFL block size is 4k.

Example:

sim1> priv set advanced
sim1*> dd if=/vol/vol0/.snapshot/hourly.2/etc/snapmirror.conf of=/vol/vol0/etc/snapmirror.conf1

NetApp Active/Active vs. Active/Passive (Stretch MetroCluster) solution

Active / Active Controller Configuration

         In this configuration both the systems are connected to each other's disk and having heartbeat connection through NVRAM card. In the situation of one controller failure other controller takes over the loads of failed controller and keeps the operation going as it's having connection with failed controller's disk shelves.

Further details of Active / Active cluster best practices can be found in TR-3450

Active / Passive (Stretch MetroCluster) Configuration

        This is the diagram of active/active metrocluster, however the same design applies to active/passive metrocluster also except one node on the cluster is having only mirror of primary system's data.

       In this configuration primary and secondary systems can extend upto 500m (upto 100km with Fabric MetroCluster) and all the primary system data is mirrored to secondary system with Sync Mirror, in the event of primary system failure all the connection automatically gets switch over to remote copy. This provides additional level of failure protection like whole disk shelf failure or multiple failures at same time, however this needs another copy of same data and exact same hardware configuration to be available for secondary node.

      Please note that cluster interconnect (CI) on NVRAM card is required for cluster configuration however 3170 offer a new architecture that incorporates a dual-controller design with the cluster interconnect on the backplane. For this reason, the FCVI card that is normally used for CI in a Fabric MetroCluster configuration must also be used for a 31xx Stretch configuration.

Further details of MetroCluster design and implementation can be found in TR-3548

Minimizing downtime with cluster

      Although having a cluster configuration saves from any unwanted downtime however a small disruption can be sensed on the network while takeover /giveback is happening which is approximately less than 90 seconds in most of the environments and it keeps the NAS network alive with few "not responding" errors on clients.

A few points in related with this are given below:

CIFS: leads to a loss of session to the clients, and possible loss of data. However clients will reconnect the session by themselves if system comes up before the timeout window.

NFS hard mounts: clients will continue to attempt reconnection indefinitely, therefore controller reboot does not affect clients unless the application issuing the request times out waiting for NFS responses. Consequently, it may be appropriate to compensate by extending the application timeout window.

NFS soft mounts: client processes continue reconnection attempts until the timeout limit is reached. While soft mounts may reduce the possibility of client instability during failover, they expose applications to the potential for silent data corruption, so are only advised in cases where client responsiveness is more important than data integrity. If TCP soft mounts are not possible, reduce the risk of UDP soft mounts by specifying long retransmission timeout values and a relatively large number of retries in the mount options (i.e., timeo=30, retrans=10).

FTP, NDMP, HTTP, backups, restores: state is lost and the operation must be retried by the client.

Applications (for example, Oracle®, Exchange): application-specific. Generally, if timeout-based, application parameters can be tuned to increase timeout intervals to exceed Data ONTAP reboot time as a means of avoiding application disruption.

Video: Enterprise Scalable NAS

Jeff Day, Marketing Manager from HP's StorageWorks division discusses why users should consider scalable NAS -- a next generation NAS technology. In this video, Jeff talks about what's driving the need for this new architecture, why it supersedes older NAS architectures and how HP is addressing this opportunity.

Run time: 15:53

Click video player to begin

NDMP (Network Data Management Protocol)

NDMP (Network Data Management Protocol) is an open protocol used to control data backup and recovery communications between primary and secondary storage in a heterogeneous network environment.
NDMP specifies a common architecture for the backup of network file servers and enables the creation of a common agent that a centralized program can use to back up data on file servers running on different platforms. By separating the data path from the control path, NDMP minimizes demands on network resources and enables localized backups and disaster

Learn More

• Storage Resources
• NAS management

recovery. With NDMP, heterogeneous network file servers can communicate directly to a network-attached tape device for backup or recovery operations. Without NDMP, administrators must remotely mount the network-attached storage (NAS) volumes on their server and back up or restore the files to directly attached tape backup and tape library devices.
NDMP addresses a problem caused by the particular nature of network-attached storage devices. These devices are not connected to networks through a central server, so they must have their own operating systems. Because NAS devices are dedicated file servers, they aren't intended to host applications such as backup software agents and clients. Consequently, administrators have to mount every NAS volume by either the Network File System (NFS) or Common Internet File System (CIFS) from a network server that does host a backup software agent. However, this cumbersome method causes an increase in network traffic and a resulting degradation of performance. NDMP uses a common data format that is written to and read from the drivers for the various devices.
Network Data Management Protocol was originally developed by NetApp Inc., but the list of data backup software and hardware vendors that support the protocol has grown significantly. Currently, the Storage Networking Industry Association (SNIA) oversees the development of the protocol.

disaster recovery plan (DRP)

A disaster recovery plan (DRP) - sometimes referred to as a business continuity plan (BCP) or business process contingency plan (BPCP) - describes how an organization is to deal with potential disasters. Just as a disaster is an event that makes the continuation of normal functions impossible, a disaster recovery plan consists of the precautions taken so that the effects of a disaster will be minimized and the organization will be able to either maintain or quickly resume mission-critical functions. Typically, disaster recovery planning involves an analysis of business processes and continuity needs; it may also include a significant focus on disaster prevention.
Disaster recovery is becoming an increasingly important aspect of enterprise computing. As devices, systems, and networks become ever more complex, there are simply more things that can go wrong. As a consequence, recovery plans have also become more complex. According to Jon William Toigo (the author

Learn More

• Network disaster recovery
• Cloud computing and SaaS

of Disaster Recovery Planning). For example, fifteen or twenty years ago if there was a threat to systems from a fire, a disaster recovery plan might consist of powering down the mainframe and other computers before the sprinkler system came on, disassembling components, and subsequently drying circuit boards in the parking lot with a hair dryer. Current enterprise systems tend to be too large and complicated for such simple and hands-on approaches, however, and interruption of service or loss of data can have serious financial impact, whether directly or through loss of customer confidence.
Appropriate plans vary from one enterprise to another, depending on variables such as the type of business, the processes involved, and the level of security needed. Disaster recovery planning may be developed within an organization or purchased as a software application or a service. It is not unusual for an enterprise to spend 25% of its information technology budget on disaster recovery.
Nevertheless, the consensus within the DR industry is that most enterprises are still ill-prepared for a disaster. According to the Disaster Recovery site, "Despite the number of very public disasters since 9/11, still only about 50 percent of companies report having a disaster recovery plan. Of those that do, nearly half have never tested their plan, which is tantamount to not having one at all."

Clustered NAS: How to choose the right clustered NAS system

           What you'll learn:  Data storage managers must have a clear picture of their unstructured data and file-system environment before investing in a clustered NAS system. We'll give you a list of questions to ask when evaluating your storage requirements and the latest clustered NAS offerings.

         

          Clustered network-attached storage (clustered NAS) uses a distributed file system that runs concurrently on multiple nodes or servers. Unlike traditional NAS, clustered NAS stripes data and metadata across storage nodes and subsystems. Clustering also provides access to all files from any of the clustered nodes regardless of the physical location of the file. But how do you determine which clustered NAS system is right for you? Here are the questions you need to ask when evaluating your own data storage requirements along with the latest offerings from vendors.

Cloud as NAS :

        Using the public cloud as a NAS tier for primary storage is a much tougher sell for most IT shops than for backups or archives. But one of the emerging technologies that could start to make that prospect more palatable is the gateway that acts as a hybrid cloud storage appliance.

           The appliances supply an on-premises cache that can provide access to the most active or frequently accessed data, so latency or network or cloud outages won’t prevent users from getting needed files. Algorithms determine which data to store in the cache.

           Many of the appliances also offer data reduction technologies such as deduplication or compression to reduce bandwidth consumption and lower the fees associated with transferring data to and from the cloud. They also encrypt the data before sending it off-premises and offer extra features such as snapshots to lighten the load on backup systems.

         Several startups currently rule the roost in the NAS hybrid cloud space and typically partner with prominent cloud storage providers. They include Ctera Networks Ltd., Nasuni Corp. and StorSimple Inc. Nasuni makes a software-based virtual NAS appliance that installs on a virtual machine (VM).

          Another option is Nirvanix Inc.’s CloudNAS product, which can transform Linux or Windows servers into a virtual NAS gateway to the company’s Storage Delivery Network (SDN) encrypted off-site storage. Nirvanix uses standard protocols such as NFS, CIFS and FTP for access to its service.

           Rick Villars, vice president of storage systems and executive strategies at Framingham, Mass.-based IDC, predicted that major NAS vendors such as EMC or NetApp will eventually provide the protocol support for a cloud tier in addition to their SSDs and SATA and SAS drives.

           “We think that day is coming. It may not be this year. It may be parts of next year,” Villars said, acknowledging the business model challenges for the NAS vendors. “That’s the last step. That hasn’t happened yet, but there’s certainly no reason why they can’t. It would require some software. It would require some links. But you could absolutely add that function in.”

Object-based storage :

      Object-based storage is hardly new. EMC pushed it into the forefront in 2002 with its Centera line in an attempt to stake out a new market known as content-addressable storage (CAS). But performance issues generally relegated the use of CAS products to archives of information that rarely if ever changed, such as medical images.

               A new wave of object storage makes use of such protocols as Representational State Transfer (REST), and is gaining a second look for near-line and primary data storage -- especially in the cloud.

         “There’s no technical barrier that says you can’t use object [storage] for primary storage,” said Andrew Reichman, a principal analyst at Cambridge, Mass.-based Forrester Research Inc. “Some primary storage is not that performance sensitive, especially with files.”

        EMC now promotes Atmos for that purpose. Other object offerings include Caringo Inc.’s CAStor, DataDirect Networks Web Object Scaler (WOS), Dell’s DX Object Storage (which uses Caringo’s technology), NetApp’s (formerly Bycast) StorageGrid, and products from startups such as Amplidata, Cleversafe Inc., Mezeo Software and Scality.

       “In the long run, we could see object [as] a replacement for file storage -- just a better way to do file storage,” Reichman said.

         Object storage is attractive to cloud storage providers because of its massive scalability and shared tenancy features, especially in comparison to ordinary file- or block-based storage.

       “You have so much metadata for each chunk of data, you can lock it down more easily and move it around based on policies and change the redundancy based on policies,” Reichman said, explaining the draw for cloud providers.

Scale out NAS

      A traditional scale-up NAS box has a fixed amount of CPU, cache and drive slots. When it fills up, the customer needs to buy another device. Scale-out NAS systems appeal to organizations with huge files because of their potential for seemingly limitless expansion while still being managed as a single storage resource.

       Also known as clustered NAS, scale-out NAS originally took aim at applications requiring high throughput and high bandwidth, such as those in media and entertainment, high-performance computing, bio-informatics, and oil and gas.

     But these scale-out systems often weren’t tuned to perform well with the typical enterprise application, where EMC Corp. and NetApp Inc. held sway with their traditional NAS devices.

      Terri McClure, a senior analyst at Enterprise Strategy Group (ESG) in Milford, Mass, said scale-out NAS tended to excel in environments with fewer numbers of unusually large files rather than the large number of small files the typical enterprise has. That made them a good choice for pplications such as video streaming. But as scale-out vendors tune their systems to perform better with more I/O-intensive enterprise applications, their systems are starting to show up in more enterprise IT shops.

       Scale-out NAS got a major shot in the arm late last year when EMC acquired Isilon Systems. Isilon offers three options: its S-Series aimed at I/O-intensive smaller files, its X-Series for fewer number of large files and its NL-Series for bulk high-capacity and low-performance storage.



         Isilon’s 72000X has a maximum capacity of 10.4 PB in a single file system from a 144-node cluster. The company’s solid-state drive (SSD)-equipped S200 has a lower maximum capacity at 2 PB, but offers 85 Gbps of aggregate throughput and 1.2 million NFS IOPS in a single file system/volume from a 144-node cluster.

      Isilon claims its distributed file system-centric system was built from the ground up for scale-out storage, where  as systems that make use of a global namespace require a software layer for scale-out NAS.

      But Jeff Boles, a senior analyst and director, validation services at Hopkinton, Mass.-based Taneja Group, said the nuances of the architecture matter less to end users than the ease with which the system scales and whether multiple storage nodes can be managed as a single storage system.

         “Scale out is still very new and innovative and proprietary,” Boles said. “Because it’s not as simple of an operation as building a controller head on an array, you’re not going to see a convergence of technologies around one best architecture.”

    In addition to Isilon’s offering, other scale-out products include BlueArc Corp.’s Mercury and Titan Series Servers (which Hitachi Data Systems resells as the Hitachi NAS platform), Dell Inc.’s PowerVault NX3500 with a clustered file system acquired from Exanet, Hewlett-Packard (HP) Co.’s X9000 family (based on technology acquired from Ibrix), and IBM’s SONAS. NetApp has a cluster mode version of its Data Ontap 8 operating system (but not a clustered file system) and Quantum Corp.’s StorNext and Symantec Corp.’s FileStor are clustered file systems that run on hardware appliances.

         Greg Schulz, founder and senior advisor at StorageIO Group in Stillwater, Minn., said some scale-out NAS products increase the number of nodes for parallel performance or large sequential streaming, while others optimize for concurrent access of multiple small random file or page views. Some focus on data storage capacity, and others emphasize clustered file systems or clustered nodes, he said.

      More scale-out options are on the way. Dell, for instance, plans to use Exanet technology to add scale-out capabilities to its EqualLogic and Compellent SAN systems, according to Scott Sinclair, senior manager of Dell enterprise storage.

          NetApp’s Brendon Howe, vice president and general manager of the NAS business unit, added via email that the company’s next-generation Ontap 8 Cluster-Mode is designed as a scale-out version of its unified architecture that extends to enterprise applications and virtualized data centers.

       “We find that segmenting the scale-out discussion to just ‘NAS’ isn't that meaningful to customers,” Howe said.

        Randy Kerns, a senior strategist at Evaluator Group Inc. in Broomfield, Colo., said although there are situations where scale-out NAS makes sense, there are also plenty of use cases where customers will prefer simpler traditional NAS.

     “It may boil down to there’s a place for both,” Kerns said. “I think scale-out NAS and traditional NAS will both be around a long time.”

Scale-out NAS, object storage, cloud gateways replacing file storage

        The insatiable need for file-based primary data storage is propelling three technologies -- scale-out network-attached storage (NAS), object-based storage and the cloud as a NAS tier -- to the forefront as potential lifelines for IT shops overwhelmed by unstructured data.

     Scale-out NAS systems can boost capacity, performance and availability with the addition of storage nodes or x86 servers equipped with a special operating system and storage. The most scalable of the clustered storage systems have the potential to manage petabytes of data across more than 100 nodes, but they’re accessed and managed as a single system through the use of a distributed file system or global namespace.

Object-based storage systems are another promising alternative to traditional NAS. Object storage foregoes traditional file systems, which have capacity and management shortcomings. Instead, these systems assign a unique identifier, or digital fingerprint, to each file plus its metadata. This identifier renders the physical location immaterial and provides massive scalability.

Using the cloud as a NAS tier is another option for IT shops coping with a flood of unstructured data. In particular, a lot of attention is gravitating toward a new wave of file-based gateway appliances that move data to a cloud service provider. These can be hardware or virtual appliances, and they can solve security and data access issues that make IT shops hesitant to use the public cloud.

Here’s what you need to know about these three NAS technologies as you plot out your file storage:


 scale-out NAS

object-based storage

cloud as a NAS