Editor's note: This is the first in a series of five articles on the 2003 Storage Innovators from the Storage Management 2003 conference on April 10 in Chicago. St. Vincent was selected as the grand prize winner of more than 30 entries.
When facing an uncertain medical future, it's not uncommon for patients to ask for a second opinion. When facing an uncertain backup and recovery future, senior engineers at St. Vincent Hospital, Andy Porter and Rich Banta, not only asked for a second opinion -- but a third and fourth.
As the largest health care provider in Indiana, St. Vincent Hospital serves more than 400,000 patients each year and the main campus supports more than 80 facilities throughout the state. This massive infrastructure generates a huge amount of data.
Porter, Banta and their team currently manage approximately 25 terabytes (TB) of data and 300 NT, Novell and Unix servers. They are seeing yearly storage growth in the range of 30% to 40% a year, amounting to about 8 TB to 12 TB. St. Vincent is also doubling growth in Intel- based servers each year.
In order to rein in some of the growth in hardware, St. Vincent turned to networked storage. About 16 TB of the environment is networked, with 10 TB on a XIOTech storage area network (SAN) and about 6 TB on a Compaq EMA 1600 SAN. The rest of the storage resides on direct-attach storage.
While the SANs helped consolidate the data center and improve management, there was still one big lingering illness in the environment -- backup.
Porter says full hospital backups were taking an exorbitant amount of time, about 36 to 48 hours for each full backup. The data was being written to tape using a Legato Networker version 6.xx and sent to a StorageTek L700 with 9840 drives. Each full backup used about 90 tapes, costing the hospital about $7,500 dollars per full backup. Another issue was that only the Intel servers were benefiting from the Legato software.
The hospital was also working on a 45-day data retention policy, but wanted to push that up to a 60-day rate.
Under the current infrastructure and factoring in the growth rates, there was no way the hospital's systems would have been able to keep up with future growth.
"We decided from the outset that conventional host-tape backup/recovery methodologies were no longer viable, and could never scale to meet our future backup/recovery needs," said Porter. "Conventional host-tape methodologies couldn't integrate well with our short-recovery time objectives, disaster recovery and high-availability requirements."
Also looming for the hospital -- the Health Insurance Portability and Accountability Act (HIPAA) regulations that are going into effect. HIPAA lays out stringent rules to protect and store patient data.
Taking these factors into consideration, Porter and Banta knew they had to come up with a plan to scale for the future. They stepped back from their day-to-day duties and devised a series of nine must-have requirements for their new backup system.
"We wanted to leverage the new ATA-based, high-density storage technologies and the new disk-to-disk-to-tape technologies (DDT)," said Porter. "[We only wanted to use] tape as an after thought for smoke and rubble recovery and archival purposes."
So, to articulate to vendors exactly what they wanted, they attended an industry conference where they sketched out their dream infrastructure on a cocktail napkin. They ruled out all vendors that were "pushing recycled host-tape technologies with DDT marketing slicks."
After looking at a number of options, they discovered EVault's InfoStage software and BladeStore from StorageTek. Picking the technology was the easy part-- selling the technology to what Porter describes as "old-school" management was the real challenge.
Porter says his storage department really had to undergo a "data driven sales campaign" to win management over. And management should be happy they did.
As a result of their due diligence, St. Vincent was able to shrink their backup window from 36 hours to 48 hours to about five hours. The implementation of disk-based backup saved the hospital $1.1 million in tangible line-item expenditures as opposed to just using tape. The reduction in backup window also did wonders for their network bandwidth. So much so that redeployed an entire Nortel 8600 switch to another part of the environment. The reduction on backup eliminated the need for a full-time backup administrator, as well.
It seems the future of backup and recovery is now well taken care of at the hospital. After the litany of savings, the only second opinion St. Vincent Hospital needs now is where to spend the extra cash.
What do you think of the Storage Innovators e-mail? Take our quick survey and let us know.
For more information on St. Vincent Hospital take a look at their Web site
>> Best Web Links: Backup
This was first published in April 2003