|
ABSTRACT
The Electrical Engineering department at Stanford University has often spearheaded the use of new technology via its collaboration with the industry or through its own research. Often, though, major changes in technology may drive an entire shift in the infrastructure required for the research and education activities of the department. The explosion of cheap, readily available storage has required us to adopt new strategies and technologies to manage data, whether its the primary storage or backup and recovery services. Foremost, as storage itself gets commoditized, we need to avoid expensive solutions and creatively adopt the next wave of solutions at minimal costs to keep abreast of what appears to be constant storage growth. Such standbys as tape-based backups may not be dead, but as will be shown, the realities of data growth require us to move beyond previous data management mechanisms. This white paper covers our use of new growth-oriented filesystems with snapshot capabilities, iSCSI for network layer independence, and a collection of other technologies provided by various vendors and open source projects to create a multi-tiered storage solution with self-service data restoration, long term growth, and disaster recovery.
In the not so distant past, research groups would tend to acquire their own infrastructure as funding sources, grants, and donations would warrant. This hardware generally included a localized file server for the group, or enough combined storage among the various hardwire networked workstations to provide for the group. Those drives were large for that time, upwards of 4.3 or 9GB at most, and most were directly used without any form of RAID or redundancy of storage. Backups generally could be run centrally, collecting from each file server the relatively trivial megabytes to potentially a gigabyte of delta change in data per day. Even with multiple research groups, incremental backups combined would average to less than 20-50GB per day at most, fitting easily in both the time window to perform the backup, as well as the average tape capacity of DLT or other media.
Within the past few years, storage use, driven by email, multi-media, and burgeoning application and file format sizes, has dramatically grown. Aiding this is the constant march to greater yet cheaper storage form factors. Where as 5 years ago, UNIX servers got by with 2-9GB, we now find workstations with a minimal of 160GB drives, and servers utilizing RAID arrays of multiple 73GB (SCSI/FC) or now 750GB (SATA) drives. Storage has also taken flight, with much data now on mobile computing platforms. Now, each group has local systems with over 200GB and servers holding terabytes of data. They still want those nightly backups, and even worse, they want further granularity. The task of backing up such data within the provided windows is arduous enough. The thought of actually recovering files, directories, or systems from multiple tapes within an acceptable time frame is almost impossible to reckon.
The solutions have already been presented by many vendors in vertical silos. We have centralized solutions of some flavor or another, including SANs, NAS, virtual tape libraries, NDMP, etc. We have even adopted some of the best in class of these, pooling money as best we can to adopt such things as Network Appliance's Filer. The advent of self-restore through regular file system snapshots has greatly aided in our perceived abilities to keep up. However, storage needs continue to grow and the cost of maintaining such products as the NetApp over many years gets to the point where we spend money feeding a low-terabyte count beast every few years for the cost of buying 10 times that storage outright in a new chassis. There is also the need to manage multiple silos of storage, and add or take away hardware without requiring clients to realign their mounts. A strategy to embrace the economics of storage pricing over time, shift storage to generic commodity units, provide advanced features such as snapshots and single logical pools of storage, and encourage the use of open standards for longevity of data access and growth is required, and we feel we've figured out the pieces.
|
