To understand how this works, let's first take a look at this from the perspective of a company that has data centers and ROBOs in different geographic locations. In this scenario, the company may want the flexibility to leverage its primary data center to recover the data backed up at any of its secondary data centers or ROBOs.

To deliver on this ideal, the company would first need to install a HYDRAstor grid at each of its data centers or ROBOs that acts independently of the HYDRAstors at the other locations. Each local HYDRAstor would then deduplicate all data at that site to address that site's need to shorten backup windows and provide fast recoveries of data at that site.

 HYDRAstor's optional RepliGrid feature then enables the company to protect data from any of its secondary data centers and ROBOs at the primary site. Using this feature, each HYDRAstor at a remote site would replicate its deduplicated data asynchronously back to the HYDRAstor at the main data center on a regularly scheduled basis. The data replication process occurs as follows:

• The HYDRAstor at the remote location provides a list of hash keys for its changed or new data to the HYDRAstor at the primary site.
• The HYDRAstor at the primary site receives the list of hash keys and removes keys it already has from the list
• The reduced list of hash keys detailing what deduplicated data is needed at the primary site is transmitted back to the HYDRAstor at the remote site
• The HYDRAstor at the remote site sends the deduplicated data associated with the hash keys in the reduced list back to the HYDRAstor at the main site

The distinct business benefits that the HYDRAstor RepliGrid feature offers are two-fold:

• Reduces Storage Costs. The amount of deduplicated data stored on the HYDRAstor at the main data center is minimized. By performing global deduplication across all local data and remote data, only net new unique deduplicated data from each remote site is stored on the main HYDRAstor.
• Reduces Network Bandwidth Costs. Since the main HYDRAstor aggregates data across all of the sites, there is a good possibility that the deduplicated data already exists at the main site. By only transmitting net new unique deduplicated data, it minimizes the amount of data transmitted and hence the size of the network pipes needed to transmit the data.

However, offering deduplication and replication is only part of the enterprise data protection picture. As companies look to deduplicate data across their entire enterprise and replicate between multiple sites, spikes in performance and capacity may necessitate the companies to have the flexibility to cost-effectively and easily scale these components of the HYDRAstor architecture. I'll examine how the HYDRAstor accomplishes those tasks in a forthcoming blog entry.

• The need to move data offsite after their data is backed up to disk. Should some type of disaster strike their production site (floods, hurricanes, tornados, power outages, etc.) they need to have some means to recover their data offsite. This generally means using either removable media (tape, optical, etc.) or replication to move it to another location.
• Companies want to use removable media as a last resort to recover data as they view it as problematic to manage and recover from. If companies do store data on removable media, they prefer to view it as data's final resting place and not their primary or even secondary means of performing a data recovery.
• Broadband network connections continue to remain affordable. The capacity and cost of a broadband connection is determined by the amount of data that a location needs to transmit. Remote and branch offices can obtain DSL (digital subscriber lines) that provide connection speeds at up to 512 Kbps for under $100/month while T1 lines (1.5 Mbps) run around $1000/month and T3 lines (45 Mbps) are about $10,000.
• Using deduplication makes replication more affordable. Rather than replicating all of the data created by each backup, using deduplication companies only need to transmit net new blocks of data. This reduces the total amount of data that companies need to send which translates into the need for smaller, lower cost network pipes for replication.

Of course, implementing replication in large enterprises as part of their overall disk-based data protection plan becomes more complicated. While large enterprises are always looking for solutions that are easy to deploy and implement into their environments, these are only some of their considerations when selecting a solution. Large enterprises may have offices and data centers that span the globe, multiple backup software products, different data retention policies and numerous network links of varying capacities. As a result, they will want a solution that they can install that matches the size and cost constraints of these different enterprise environments.

Enterprises also need to identify products that provide architectures that meet current and future needs. Features such as global deduplication and more granular control over replication, such as what data is replicated and when it is replicated, are almost prerequisites in enterprise shops. These companies also need to think outside the box to take into account new architectures such as grid storage that provide new enterprise data recovery options. For instance, using grid storage, companies can start to think about creating storage grids that replicate data across geographic distances such that companies can recover data from any of the sites.

End-users now readily recognize that replication is becoming as integral to their corporate data protection plan as data protection software and hard disk storage systems. However enterprise corporations need to think more broadly about how they implement replication as part of their disk-based data protection strategy.

Deduplication and replication alone are not enough as large enterprises require disk-based backup solutions that scale in a manner that is very different than what is generally available on the market. These new solutions require new architectures such as grid storage that scale in such a way that they remain cost-effective short and long term. The grid storage architecture the makes up NEC HYDRAstor provides the type of scalability and cost effectiveness to complement the deduplication and replication features that large enterprise companies are seeking. In an upcoming blog entry, I'll take a closer look at how the HYDRAstor delivers on these features.

I first met Mr. Paulk at the Fall 2007 Storage Decisions conference in New York City and interviewed him shortly thereafter for a blog entry that appeared back in October 2007. However six months have passed since that interview so I followed up with Mr. Paulk to get an update on how his installation of the NEC HYDRAstor was performing, since Paulk was still using a beta version of the NEC HYDRAstor software when we last spoke.

Paulk revealed that he is now in full production with the production code loaded on the NEC HYDRAstor. However he is still using the same hardware configuration (two Accelerator Nodes and four Storage Nodes) that he started out using due to the high deduplication ratio that he is achieving with the HYDRAstor.

Last fall he was achieving a 17:1 deduplication ratio and hoped to eventually achieve a 35:1 ratio. Six months later, his deduplication ratio is now approximately 39:1 which has mitigated his need to buy additional capacity and has driven his cost/GB down to approximately 70¢/GB. "It's like getting 390 TB for the price of 10 TBs," says Paulk.

He also has not found it necessary to add more Accelerator Nodes into his HYDRAstor configuration. Though he has nearly doubled the number of servers he is backing up on a nightly basis (from 13 to 21 servers), he is achieving about 3 Gbps of throughput across his two Accelerator Nodes.

I then asked him, "What are the biggest benefits that you have experienced since you started using the HYDRAstor?" There were four benefits he cited:

• First, it worked as advertised. The installation was easy (it took 68 minutes), and it has done everything he has needed it to do.
• Second, it requires very little management overhead. He has one individual assigned to manage the HYDRAstor and, since it functions as one logical configuration, it takes very little time to manage.
• Third, no backups have failed since he introduced the HYDRAstor, and it works 90% faster than when he was using tape.
• Fourth, he has found it has reduced his stress level. Aside from alleviating his backup concerns, the HYDRAstor provides him a solid foundation that he can use to build for the future. He no longer has the traditional worries of how he will manage, upgrade or migrate data to new storage systems, because the HYDRAstor accounts for all of these concerns with its grid storage architecture.

In the next few months, Paulk plans to archive about one million documents to the HYDRAstor, which will consume about another 1.4 TBs of storage. What he is curious to discover is how they will impact his deduplication ratio. These one million documents are currently paper documents that need to be scanned so he wonders how they will impact the level of deduplication that he is seeing with his backup data once they are stored on the HYDRAstor. My plan is to catch up and speak with Mr. Paulk again this fall so DCIG can share some more of his story and experiences at that time.

• Self managing. NEC's HYDRAstor architecture supports self-management through the dynamic addition of nodes (servers) that offer either more capacity or performance. As more Accelerator or Storage Nodes are added to the HYDRAstor grid architecture, it non-disruptively redistributes data across old and new nodes to optimize performance and maximize data resiliency. This eliminates the normal processes of provisioning, sizing and data migrations that administrators have to perform, while alleviating the management overhead and costs associated with archive and backup processes.
• Data mobility. Data mobility comes more prominently into play when new Storage Nodes are added into the HYDRAstor grid storage architecture as well as during technology refreshes of Storage Nodes. As new Storage Nodes are added, the HYDRAstor re-balances data across existing and new Storage Nodes to simplify data management. As existing nodes are retired, data from an existing node is automatically migrated to a new node and, if companies have multiple sites, HYDRAstor supports the movement of data to alternative sites to create an enhanced level of data resiliency.
• Non-disruptive evolution. The HYDRAstor grid storage architecture addresses one of the most problematic aspects of storage management today: technology refreshes. As current storage systems age, usually the only option companies have is to purchase an entirely new storage system and then use either host or network based data migration tools to move to new storage controller architectures. Since HYDRAstor is based on a grid architecture, it can transparently evolve to newer technology simply through the introduction of new Accelerator or Storage Nodes based on the latest and greatest hardware technology. The HYDRAstor adds these new nodes into the grid while older nodes are marked for decommissioning and non-disruptively taken out of service.
• Scalability without trade-offs. A key problem with current storage system architectures is that you generally have to pick between performance, capacity and cost when scaling the architecture. Since HYDRAstor's Accelerator and Storage Nodes are based on industry-standard, off-the-shelf hardware, the typical hardware costs associated with proprietary storage hardware architectures are avoided. Since HYDRAstor also decouples performance (Accelerator Nodes) and capacity (Storage Nodes), users can scale one or both to accommodate whichever direction their storage environment grows.
• Enhanced, flexible resiliency. The HYDRAstor accounts for the growing possibility that today's RAID data protection architectures are insufficient when deduplication is used across 100s or 1000s of TBs of capacity. Administrators can define the level of data redundancy that is appropriate to their site and the HYDRAstor will dynamically distribute the data across the nodes to deliver the desired level of resiliency.
• Integrated data management services. The HYDRAstor comes with two sets of integrated data management services. The base level of services includes the automated management of the data, more efficient storage (deduplication) and enhanced data resiliency. Advanced services like replication, WORM, security, classification and search are other features that users can optionally license from NEC.
• Industry standard interface support. The HYDRAstor presents an industry-standard NFS and/or CIFS interface, so any Linux, Windows or UNIX server can archive data to it or, alternatively, any backup software can treat it as a disk cache and store data on it.

Compelling statistics, no doubt, and while that report translates into a lot of good news and healthy revenues for storage vendors, storage managers grappling with this growth in information are probably less than enthused by this report. As most storage managers know, storage growth no longer automatically equates to more personnel to help in the task of managing this data and the underlying storage systems.

It wasn't that long ago that most enterprises followed a general rule of thumb in terms of how many TBs of storage that a storage administrator could manage. Whether that ratio was 1:1, 10:1 or 20:1, as more TBs of storage were added, storage managers had some assurance that they would receive more storage administrators or more robust storage management tools to help them manage this additional data.

This traditional assumption of "more storage=more people" is no longer realistic. Storage systems with inexpensive, high capacity SATA disk drives are now routinely used in the storage of archived and backup data. While these forms of data typically do not require the same level of day-to-day oversight and management that production data requires, managing the data on these storage systems takes on a different nature.

Responsibilities such as managing capacity and performance growth, replicating data offsite for DR, copying or backing up data to removable media such as optical or tape and migrating data during technology refreshes can grow in magnitude as the amount of storage grows. Toss in the time required to stay up-to-date and best manage new technologies like deduplication and it becomes obvious that bringing in these new storage systems is not necessarily a plug-n-play proposition.

However there is an added twist to the ongoing management of these storage systems for this data. These storage systems are viewed as operational expenses and not critical to daily business operations so no new staff is added for their ongoing management. As a result, the responsibility of managing these storage systems typically falls to existing individuals who try to manage these systems as they have time.

These problems are creating the need for a new type of storage system architecture that is specifically designed to address these needs. NEC's Vice President of Advanced Storage Products, Karen Dutch, addressed some of these concerns in her recent Spring 2008 SNW presentation, "Defining Storage Solutions in the Data Center 2.0". In this presentation, she provided a number of pointers as to what features these new storage architectures should deliver to keep storage management in this environment manageable including:

• Self managing
• Data mobility
• Non-disruptive evolution
• Scalability without trade-offs
• Enhanced, flexible resiliency
• Integrated data management services/li>
• Industry standard interface support

In the next blog entry, I'll take a closer look at these points and how NEC's HYDRAstor is delivering on these new objectives.

The benefits that either a grid storage or a siloed, two controller storage system will provide in terms of deduplication will hinge on the amount of data that a company plans to store on the system. When there is only a small amount of data (under 10 TB), Dr. Toelg says there is probably not a big difference in the deduplication benefits that a company will realize when using one approach over another. In these circumstances, the other benefits of using a grid storage architecture, such as its ease of upgradeability and "future proofing" against technology obsolescence, become the main drivers for its adoption.

It is in enterprises that need to store tens, hundreds or even thousands of terabytes of data that the drawbacks of a siloed, two controller architectures become more apparent. These architectures force enterprises to deploy multiple appliances that create data silos. Multiple appliances negate benefits of deduplication since the second, third and future appliances store similar data as previous appliances and do not deduplicate across one another. "As you add more appliances, the effective deduplication rate drops from 18:1 - 20:1 to 8:1 - 9:1 which thereby cuts your total effective deduplication capacity," says Dr. Toelg.

NEC HYDRAstor's grid storage architecture can deliver a higher deduplication ratio since its Storage Nodes create one logical pool of storage capacity and globally deduplicate data across that entire pool. The actual deduplication ratio any enterprise gets will vary according to how long data is retained, the data or application type since backups will likely get better dedupe ratios than archive data, the type of backups performed (full, incremental, or differential) and the uniqueness of the data stored on the appliance. However, the odds of achieving higher deduplication ratios are improved using HYDRAstor since companies can store all data in one logical pool.

The grid storage architecture also gives enterprises a couple of other advantages. Since all Accelerator Nodes can access data on any Storage Nodes, companies that plan to continue their use of tape can dedicate certain Accelerator Nodes to transfer data from disk to tape. This configuration prevents the performance overhead that data migrations incur from overlapping and impacting production backups and restores. Companies that run backups and restores 24x7 are the most apt to want to take advantage of this option.

The other notable advantage that a grid storage architecture provides is that it keeps the footprint of the system to a minimum. Since HYDRAstor can scale Accelerator Nodes and Storage Nodes independently, enterprises only need to bring in as many nodes as they need and upgrade or add either capacity or performance at any time. This flexibility minimizes power consumption, data center floor space and the cost of HYDRAstor since companies only need to purchase as many Accelerator or Storage Nodes as they need, when they need them. HYDRAstor's global deduplication feature further contributes to keeping the hardware footprint to a minimum, by improving the overall deduplication ratio and reducing the number of nodes required to hold and serve the data.

NEC HYDRAstor's grid storage architecture simplifies deduplication while maximizing its benefits. Companies can start with the HYDRAstor in configurations that match their initial requirements. They can then scale it according to how their environment evolves without sacrificing higher deduplication ratios or ease of management. HYDRAstor's grid storage architecture sets the stage for companies to confidently deduplicate their data while avoiding the problems that siloed, two-controller architectures introduce.

Jerome: How do the HYDRAstor Accelerator and Storage Nodes differ from one another in their hardware?

Dr. Toelg: For the most part, they use off-the-shelf hardware, though there is one big difference between them. Accelerator Nodes do not need to store large amounts of data like the Storage Nodes do. Since the Accelerator Nodes only need enough storage for the operating system, they just use standard disk drives that are mirrored. Conversely, we want to put as much storage as possible into the Storage Nodes. Currently these nodes use 500 GB SATA disk drives, though 750 GB and 1 TB SATA disk drives for the Storage Nodes will be available in the near future.

Jerome: Since their hardware is similar, how do the nodes differ in the software that operates on them?

Dr. Toelg: The Accelerator Nodes use file systems to present NAS interfaces to the connecting servers; backup clients see the NAS interface presented by the Accelerator Nodes and the Accelerator Nodes see the NAS interface presented by the Storage Nodes. However, what the software on these nodes does under the covers is very different. The Accelerator Nodes pre-process incoming data by chunking the data and then doing de-duplication of the data. The primary function of the Storage Nodes is to protect the data and then assign the data to the nodes where it is most efficiently and effectively protected.

Jerome: So both the Accelerator and the Storage Nodes deduplicate data?

Dr. Toelg: That is correct. HYDRAstor uses a two-step inline process to deduplicate data. Two or more Accelerator Nodes may see the same file at the same time. However, Accelerator Nodes only have a part of the information required to do deduplication and do not maintain the entire global deduplication index. So the Accelerator Nodes chunk up each file into small chunks, eliminate as many duplicates as possible and then send the remaining chunks to the Storage Nodes. The Storage Nodes receive these chunks of data and then make the final determination as on which chunks are unique and should be stored to minimize storage requirements. Data is protected by breaking the unique chunks of data up into fragments and distributing the fragments across the Storage Nodes.

Jerome: How do the Storage Nodes protect the data without impacting performance?

Dr. Toelg: HYDRAstor's two-tier grid architecture has been designed so that many read and write processes can run in parallel by splitting tasks between Accelerator and Storage Nodes and distributing the workload across many nodes. This makes the HYDRAstor unique when compared to monolithic appliances as it allows you to scale performance and capacity in parallel. Furthermore, the HYDRAstor system is constantly monitored for the discovery of new nodes, component failures or the removal of specific nodes from the cluster of Storage Nodes. Background tasks ensure the system is balanced with respect to storage capacity and performance at all times without manual interaction. By distributing tasks so read and writer performance is maximized and maintenance of stored data such as migration of data to optimize utilization, data deletion or recovery of lost data is ensured.  

• Why data silos are created
• The problems that data silos can create

The primary reason that many deduplicating appliances create data silos is that they are based on the traditional dual-controller storage system architecture. Dual-controller storage systems typically use two clustered servers that sit in front of a fixed pool of storage. These two servers provide high availability, improved performance and access to the data stored on backend pool of storage by either server.

Traditional storage systems serve consumers well when storage growth is limited. But this model starts to break down when used in conjunction with deduplicated backup data stores. Anecdotal evidence suggests that data growth in most companies continues to grow at rates of 50% or more every year. Using deduplication in the backup process does significantly reduce the amount of data that companies need to store. However, over time, even deduplicating backup appliances eventually need more performance and storage capacity and the underlying architecture of traditional dual-controller storage systems does not permit them to scale.

Companies then need to bring in more deduplicating backup appliances with isolated backend storage in an attempt to keep up with data growth. Yet when that occurs, the new appliance is unaware of the first appliance and is unable to take advantage of any of the indexed deduplicated data stores that it has created. As a result, the second appliance must start from scratch and build its own deduplicated data store which creates the new data silo.

Data silos create problems on a number of fronts. More appliances create more points of management. Backups become more difficult to manage since administrators need to determine what backup jobs they need to send where. Send too many backup jobs to the new appliance and it may encounter the same performance or capacity problems as the existing appliance. Send too few jobs and the new appliance will not deliver the full benefits of deduplication. In addition, separating the backup jobs to isolated deduplicating appliances with different data stores inhibits the ability to deduplicate data globally across all backup jobs.

These types of issues create a demand for a new type of storage architecture with global deduplication capabilities that new products based on grid storage architectures can address. NEC HYDRAstor is one such product that uses a grid architecture to allow it to scale performance and capacity independently. Companies can start with a configuration that meets their immediate data storage needs and budget restrictions. However companies can scale it without needing to create additional points of management or new data silos.

Because the NEC HYDRAstor functions as one logical storage system regardless of how much performance or capacity a company adds, it can globally deduplicate all company archive or backup data stored on it. This reduces and optimizes corporate data stores while eliminating the headaches associated with managing multiple appliances and backup jobs.

The adoption of disk-based backup and deduplication is accelerating in more companies. As it does, new storage system architectures are needed to help companies solve the type of problems that deduplication and today's data growth creates without creating data silos. Products based on grid storage architectures such as NEC HYDRAstor can help eliminate data silos while giving companies newfound flexibility to manage their growing volumes of data while minimizing the amount of data they need to store.

In fact, there was a great debate last fall at SNW between some of the deduplication vendors as to which way was the best way to deduplicate data. While entertaining, this debate solved little. Most users only care about deduplication insomuch as it affects their ability to successfully backup the data in their environment. However most of the focus in these discussions focus on solving their short-term requirements and don't take into consideration some of the longer term problems. Unfortunately hidden long term data management costs lurk for those who unwittingly adopt the wrong deduplication appliances.

Deduplicating appliances have gained mindshare with users because it makes disk as cheap, or cheaper, than tape by delivering data reduction ratios of 15:1 or more while expediting backups which solves their short term backup problems. However companies also need to consider, when selecting a deduplication product, how well it will best serve them in the long term.

For example, does the deduplication appliance extend the benefits of deduplication beyond the current appliance and, if it does, how does it do it? The capability to globally deduplicate data is very powerful, but most deduplicating storage appliances are limited in scope to just that one appliance. If the ceiling for performance or storage capacity is reached, one must bring in a new appliance. Thus, deduplication starts from scratch, even if the data on the initial appliance(s) have been deduplicated. The impact is silos of deduplicated data.

Is there an adverse impact of deduplicating silos of data annually? That really depends, but companies I talk to are experiencing year-over-year data growth rates of 50% or more. Recently, I spoke to a colleague at my previous employer and he said the storage rate there has continued to double yearly. While we believe data growth rates will slow down, anecdotal evidence suggests that it is not.

I bring this example with my previous employer up because as disk prices continue to drop, companies are bringing in more storage to store more data, often to make copies of existing data for other purposes such as testing, development, data mining and eDiscovery. But as companies back this repurposed data up to new appliances, they can not take advantage of the deduplication benefits that existing appliances provide.

Siloed deduplication offered some benefits during initial use, but forward looking companies realize global deduplication fully capitalizes on deduplication technology to provide the lowest TCO solution that meets business data requirements. Few products are available to support global deduplication, but new products like the NEC HYDRAstor can deliver this based on its storage architecture. Since HYDRAstor is based on a grid storage architecture, it can scale performance and capacity independently allowing IT to deploy one system instead of many. This architecture enables it to globally deduplicate data without running up against the performance and capacity limitations that current appliance based products encounter and force IT to deploy instances, which can easily create a management nightmare. Global deduplication also lowers total storage costs since the amount of data stored is minimized.

Using deduplication is cost-effective justification used to introduce disk into the backup process. However companies must recognize that in order to fully leverage the benefits of deduplication and keep it from becoming just another management pain point a few years down the road, they need to look beyond just the problems deduplication solves and what new problems it creates when you have many silos of deduplicated data. As companies consider their mid-to-long term business issues, emerging features like global deduplication and grid storage architectures merit special consideration in the data deduplication management process.

Introducing disk as a target into the backup process changes this scenario. Companies must now begin to account for the difficulties that migrating data from disk to tape or other disk systems once the initial backup completes. The magnitude of the problem varies according to how the disk is configured. For instance:

• Using disk-as-disk may sound like the simple solution but if users are backing up vast amounts of data, the storage system may continually run out of storage capacity. This results in failed backup jobs and users needing to continually procure more storage space.
• Virtual tape libraries (VTLs) are becoming popular in enterprises because they look and act like physical tape libraries to backup software while providing the backup and recovery speeds of disk. However migrating data from virtual tape cartridges to physical tape cartridges is fraught with problems. Companies need to keep backup catalogs in sync, account for compression and encryption on physical tape drives and account for differences in size between virtual and physical tape cartridges.
• Deduplicating backup appliances appear as the perfect technology for backing up to disk since it eliminates redundant data, can increase storage capacities by 10 fold or more and gives the appearance of infinite capacity. There's the rub. Deduplicating backup appliances only give the appearance of infinite capacity so they may require upgrades or migrating data to tape. In either case, companies need to plan and manage the migration.

So while the initial benefits that companies derive from using disk in any of its different formats in the backup process are usually substantial, the effort associated with managing and migrating backup data to support a technology refresh or disk system expansion to migrate from disk to tape over time can become problematic.

That's what makes the NEC HYDRAstor truly unique among disk-based data protection options. Even though it falls under the general category of a "deduplicating backup appliance", because it can scale to manage petabytes of data while concurrently scaling performance, companies do not need to migrate backup data for disk to tape or upgrade to a new appliance when they are running short on capacity. The NEC HYDRAstor eliminates these problems by decoupling the scaling of performance and capacity servers, or nodes, so companies can scale either to meet the particular needs of their environment without replacing existing equipment. The HYDRAstor also helps companies avoid the typical need to do data migrations during technology refreshes since data on existing nodes can be non-disruptively migrated to new nodes without downtime.

The introduction of disk into the backup process has fundamentally changed the backup process but with that change in process companies also need to change how they think about managing backup data once it is stored to disk. Most disk-based products used in the backup process have not fully accounted for this fundamental change in the management of backup data after it is stored on disk. The NEC HYDRAstor is one such product that has taken the management of backup data stored on disk into account in its design from the outset and prevents this from becoming a problem later on in the management of the data.

Though HYDRAstor is intended as a possible replacement for tape, it does not need to replace all tape immediately... or ever. While replacing tape with disk is probably the objective for most companies - and one that may occur over time - this will not occur overnight. In the meantime, companies need to keep data offsite as part of their comprehensive data protection plan, which for most companies means that they will still need to copy data from disk to tape.

This configuration plays to HYDRAstor's strength. It uses NAS to present a large disk target to the backup software which gives it a distinct advantage over using Virtual Tape Libraries (VTLs). In order for the backup software to appropriately track the data when copying data from a virtual tape cartridge to a physical tape cartridge, the amount of data on a physical tape cartridge needs to match the amount of data on a virtual tape cartridge. However if the virtual tape cartridge is not filled to capacity then the physical tape cartridge is not filled either.

Using HYDRAstor as a large disk pool or cache changes this dynamic. When data is copied from disk to tape, the backup software copies data from the HYDRAstor and fills up each physical tape cartridge. Even though HYDRAstor stores the data in a deduplicated state, because HYDRAstor decouples Accelerator nodes from Storage nodes in its grid architecture, companies can increase performance if needed.

Backup is not the only way that companies can use HYDRAstor. While using it as a backup target is certainly the most popular option right now, using it to store archived data or as a network file server are other ways that companies can also look to use HYDRAstor.

Data deduplication is changing the backup game but it has the potential longer term to change how companies store much of their data. It's when companies start to view deduplication from this larger viewpoint that they can begin to understand the significance of HYDRAstor's underlying grid architecture. It can adapt to meet the different ways companies need to use it in the backup process but gives companies the option to use it to meet multiple, different requirements.

Scaling a storage grid to manage and deduplicate hundreds of TBs or even PBs of data while still protecting the integrity of the data and without creating performance bottlenecks is a different animal than merely delivering grid storage. This last week I hooked up with Dr. Christian Toelg, NEC's Director of Business Development, to discuss how NEC architected HYDRAstor to meet these specific challenges.

Dr. Toelg believes that the future of disk-based data protection clearly lies with deduplication. Unfortunately deduplication does not play well using traditional clustered storage system architectures as the amount of data under management continues to grow. Traditional clustered storage systems have a finite number of controllers and storage capacity as well as centralized file system information and mapping tables on each node. Together, these factors create hard upper limits to scalability forcing end users to deploy and manage additional systems that function as isolated data silos with no deduplication across systems increasing management complexity.

HYDRAstor's grid storage architecture conversely can start small but linearly and independently scale performance or capacity or both. HYDRAstor decouples performance nodes (called Accelerator Nodes) from capacity nodes (called Storage Nodes) creating two separate layers that act independently of one another and possess their own levels of intelligence.

In traditional clustered storage architectures, the storage controllers possess all of the information about the data while its attached disk remains stupid. HYDRAstor changes this paradigm by putting intelligence in both layers and makes the Storage Nodes "self-aware". These Storage Nodes behave as one large pool of capacity that is fully self-managed with respect to balancing capacity and performance across all nodes as well as rebalancing the system as new capacity is added or to recover from failures.

Creating a separate storage layer introduces new data protection and disaster recovery options. Dr Toelg says that NEC made a specific decision early on in HYDRAstor's design not to use RAID since loosing any chunk of deduplicated data can impact tens, hundreds or thousands of file and make all of them unrecoverable. Because the storage layer possesses its own level of intelligence, HYDRAstor can create higher levels of data protection and redundancy than what RAID can deliver with a comparable storage overhead. Also, deduplicated data can be shared between storage pools in different geographical locations using policies to replicate data so companies can automate their disaster recovery configuration.

By creating two distinct layers in its grid storage architecture, HYDRAstor addresses key concerns about data protection and scalability that deduplication typically introduces. HYDRAstor eliminates the capacity, cost and performance trade-offs that companies typically need to make while laying the foundation for companies to bring a disaster recovery solution into their environment.

The biggest benefit of deduplication - high data reduction ratios - can also become its biggest drawback over time where recoveries are performed. As deduplication breaks incoming backup streams apart into smaller chunks of data, it compares these incoming chunks of data to chunks of data it already has on its storage subsystems, storing and indexing net new chunks of data while indexing duplicate chunks.

The problem that begins to emerge over time from a recovery perspective is inadequate processing power to reconstruct the files. Tens, hundreds or even thousands of files may reference the same chunks of data that are spread throughout the deduplicating appliance so it takes the system longer to reconstruct specific files. This scenario becomes especially problematic if the deduplicating appliance is called upon to reconstruct and recover multiple files for tens or hundreds of servers at the same time; such as may be required during a Disaster Recovery (DR) scenario.

NEC's HYDRAstor has a decided edge over its competitors. HYDRAstor uses a grid architecture that supports both accelerator and capacity nodes so users can scale performance and capacity independently of one another. This architecture is critical in recoveries since companies can add more accelerator nodes to meet specific application or corporate Recovery Time Objectives (RTOs).

For instance, it is more likely that a company will only need to recover files from a single server at one time so less accelerator nodes may be needed since the performance overhead is not as great. However if data is moved off to tape or they need to recover files at a disaster recovery site the company may need to concurrently recover tens or hundreds of servers in a short, fixed period of time. Using HYDRAstor, they can add additional accelerator nodes to meet these specific corporate RTOs.

Data recoveries should be first and foremost on the minds of administrators when purchasing disk-based data protection. If companies cannot recover data in time to satisfy specific application or corporate DR RTOs, its value proposition becomes questionable since data protection is ultimately about data recoveries, not faster backups or data reduction. NEC's HYDRAstor remembered to deliver on that objective.

As I was going through the job interview process for a storage administrator position nearly seven years ago, my prospective employer took me on a tour of the data center in which I was to eventually work. Having always worked in smaller data centers, this tour took me aback. Entire rooms were filled with tape libraries while other rooms were filled with racks of tapes staged for offsite delivery or prepped for someone to put them back in the tape libraries.

Yet by the time I left that company about 18 months ago, most of that tape infrastructure was gone and replaced by disk as disk became the predominant method for backup and recovery. What is of note is that my former employer financially justified the changeover from tape to disk prior to the advent of deduplication.

This is an interesting point to ponder since if deduplication had been as widely available on disk subsystems three or four years ago as it is now, would my company have bought disk with deduplication or without? That may seem like a stupid question but companies should not assume that just because a disk library supports deduplication that it is a better choice than just buying raw disk and backing up all of their data on it.

Before adopting deduplication, one needs to consider all of the intangibles. The overhead associated with deduplicating the data, the additional time needed to recover the data and the risks associated with storing data in a deduplicated state contribute to the argument that deduplication is not a slam dunk in every circumstance.

So am I saying that one should stop reading this blog about HYDRAstor's deduplication features and start shopping for disk drives at Wal-mart? Not exactly. But what I am saying is storage administrators, engineers and architects need to take a step back and take a look at the larger picture of what problems they are really trying to solve before implementing deduplication.

While reducing their data stores is one of the problems they are trying to solve, it is not the only one. The real problem that most companies are trying to solve with disk is improving their backup and recovery windows which, at the enterprise level, become a much more complex architectural challenge than just buying a storage system that supports deduplication.

This is what makes HYDRAstor an extremely compelling story. Yes, it does do deduplication and maybe even arguably better than other products. But more importantly its architecture addresses the reasons companies introduced disk-based backup in the first place - to improve the speed of their backups and recoveries. By delivering an architecture that can independently scale performance and capacity, companies can take advantage of the benefits of deduplication while at the same time providing fast backup and recovery times.

Deduplication will store data more efficiently in a smaller footprint - no one disputes that. Yet for companies who prioritize data reduction ratios above faster backup and recovery times, they will quickly be reminded that it was faster backup and recovery times, not data reduction, that was their primary motivation for switching to disk in the first place.