First American Trust Federal Savings Bank is a banker's bank. While it offers kid-glove trust and estate planning
services to high net worth individuals, the bulk of its operations are electronic banking services for high-volume businesses. Its biggest customers are sister companies in First American Corporation, a Santa Ana, CA-based conglomerate with 2002 annual revenues of around $1.5 billion. If wealthy individuals expect superb service, the exigencies of the B2B side demand it. If First American failed to execute a transaction due to a network glitch -- or anything else -- the bank would be liable for the interest lost.
"That's why we can't have things break," said chief technology officer Henry Jenkins, who has somehow learned to take the intense pressure and immense responsibility of his job in stride.
For a while, it wasn't so easy for Jenkins to remain cool. Back when First American focused its business on wealth management, things were under control -- around 110 employees handled operations that brought in around $10 million in revenue. They used Microsoft Exchange Server for e-mail, a customer relationship management application called Onyx, WinImage by halFILE for document imaging and a number of homegrown applications. The bank simply kept all its data in SQL databases on direct-attached storage (DAS); each application server typically had its own RAID.
Then in 1999, First American moved into B2B banking and the pressure was on. Revenue began to grow. By 2002, it had doubled to $22 million -- while employee count remained the same. The network needed to keep pace with the new world of e-business. First American's data ballooned, in part because it began to offer client statements online in the obese PDF format. It was time for an infrastructure upgrade.
First American installed a high-speed data center backbone, standardized the company on Windows NT workstations for desktop users, and added Windows NT and Linux servers. Most important, it moved all data to a central Network Appliance network-attached storage (NAS) component. "At this stage, we were primarily concerned about storage consolidation," Jenkins said, "especially for user files, imaging and SQL databases." The single NetApp filer provided 360 gigabytes (GB) of storage. It served user files and wire services files processed on the Linux servers, and also was a storage location for SQL Server 7 backups.
Jenkins was looking for three things: simple administration, reliability with disaster protection and the ability to add storage economically. He considered a storage area network (SAN), but found the options "horrendously expensive." The NAS solution was still relatively new, but it was nearly 20% cheaper than a SAN. Jenkins narrowed the field to products from Hitachi Data Systems, Procom and Network Appliance. "I was sold primarily on their presence on so many tech sites," Jenkins said. "They had quite a bit of market penetration." He also saw that the NAS would be easy to expand, while administration took just five minutes a week.
First American has continued to upgrade and expand its network and storage. In 2000, First American added a second NetApp filer in the company's configuration to bolster redundancy. Its current infrastructure stands at 35 Windows NT servers and six Linux servers.
"We've continued and expanded the Linux presence because we do a lot of internal development on Linux," Jenkins explained. It made sense for the company to build its own applications, such as budget estimating and fee review software. "Wealth management is very much a niche business," Jenkins said. "I don't think I could get a consulting business off the street that understands our needs."
First American's home-baked apps and reporting engine take XML code and deliver it into various output files for management, such as PDF documents or Excel spreadsheets. Each application is powered by a SQL database on the NetApp filer. Most of the reports are stored on the NAS as well, and Network Appliance's ability to handle Linux files was another important consideration.
In 2002, Jenkins also added NetApp's SnapDrive product to mirror the SQL databases, Exchange information stores and user files to another NetApp filer at a hot site in San Diego. The failover cluster sends updated information to the hot site every five minutes, and a recent test showed a lag time of only three minutes in the data.
The IT team did no formal measurements to determine what kind of return it was getting on the investment in the NAS. But there were some clear benefits. Throughput was not affected by the switch from direct attached to NAS, while online backups of the SQL databases were cut from 30 minutes to five minutes when done directly to NAS instead of to a Windows server. Most important, "We had no user complaints," Jenkins said, "and that's what I'm measured on."
This year has already seen continuing upgrades. Jenkins added more storage to the existing failover cluster. "We found out you can't have too much," he said. "We may never meet our goal of 100% uptime, but we'll always strive for it." Thanks to NAS, Jenkins has gotten close without breaking a sweat.
For more information on First American Bank, visit its Web site.
Additional information on Network Appliance can be found here.
>> Best Web Links: NAS