City of Houston Case Study

“When we compared the different possibilities available on the market and completed the demos, Data Domain came out ahead in terms of functionality, performance and cost. Data Domain worked out to be about two to three times cheaper than what it would have cost us to add tape libraries or VTL.”

Rick Nicholson
Senior Central Network Administrator
City of Houston ITD

• City of Houston Case Study (PDF)

Streamlining Data Protection Processes with Data Domain

The City of Houston is the fourth most populous city in the nation, and is the largest in the southern U.S. and Texas. Founded in 1836, it had a 2006 estimated population of 2.14 million, according to the U.S. Census Bureau. The metro area’s population of 5.54 million in 2006 is sixth largest among U.S. metropolitan statistical areas.

A large portion of the City of Houston’s data is managed by the IT Department (ITD), which was created in 2003 to improve information technology organization throughout the city, leverage emerging technologies to reduce cost and improve services to citizens and employees. The ITD operates seven locations, including two primary data centers and several remote offices. Currently, there are 700 servers in operation citywide, and about half of these are managed directly by ITD. They run Microsoft Exchange email, SQL and Oracle databases, and file and print services, most applications harnessed via direct attached storage (DAS), as well as one or two small SANs.

Customer Challenges

About 15 to 20 TB of data is backed up each week by ITD. This includes a full backup on the weekend and incremental changes daily. However, the city’s aging tape libraries were struggling to cope.

“We were using tape backup at each location and this caused constant problems,” said Rick Nicholson, senior central network administrator for the City of Houston ITD. “Many backup jobs didn’t complete, tape changers would go offline, tapes would often break and we would experience errors in the tape drives.”

In this environment, the backup staff would have to constantly baby sit tape backups which would cut into their evenings and weekends. Even with this ongoing administration, often they would arrive early on Monday mornings only to deal with a litany of backup problems. Many times the team was forced to complete any lost or missed backups before the start of the workday when servers were needed back in production.

Restores, too, were a major challenge. With every restore request, the ITD backup team had to rifle through a pile of backup tapes to find the right one. It could take several days to complete one restore.

Because its production database and many other applications central to city operations are shared between five locations, ITD was backing up many of the same data sets over and over again. The team concluded they could simplify management of the backups by centralizing its tape infrastructure, redirecting tape resources to an archive role and adding disk into the process.

“We realized that by backing up to disk we would not have to deal with the slow performance of tape,” said Nicholson. “Further, we saw the long term potential of centralizing backups to revolutionize the entire workflow and optimize our data protection.”

Data Domain Solution

The City of Houston ITD first looked into the possibility of tape technology to achieve its goals, but quickly realized that a tape fix was more expensive than a deduplication storage alternative from Data Domain, and it didn’t solve their main challenges. They also investigated adoption of conventional disk and VTL, but these approaches turned out to be even more cost-prohibitive due to the size of the system needed to meet their future objectives.

“When we compared the different possibilities available on the market and completed the demos, Data Domain came out ahead in terms of functionality, performance and cost,” said Nicholson. “Data Domain worked out to be about two to three times cheaper than what it would have cost us to add tape libraries or VTL.”

The City of Houston ITD purchased eight Data Domain deduplication storage systems – four DD510 and four DD530 appliances to protect its Microsoft Exchange, SQL database, SAP, and file and print server applications.

John Vanden Bosch, lead backup administrator for the City of Houston ITD reports that the organization purchased installation services from Data Domain and a sales support engineer implemented the first of the city’s Data Domain systems. Once accomplished, Vanden Bosch has been able to deploy the rest of the appliances on his own. This set-up and configuration process takes less than two hours from start to finish, according to Vanden Bosch.

Business Benefits

Previously, the city used many tapes each night which added to their ongoing IT expenses. Just backing up Exchange, for example, consumed about 1 TB of tape media. Weekly backups took up about 3.5 TB of tape. These ongoing operational expenses have largely been eliminated with the deployment of Data Domain systems.

“Deduplication via Data Domain means we don’t need to be constantly swapping out or purchasing tapes anymore,” said Nicholson. “We are getting a compression ratio of about 10 to one with Exchange and about 17 to one on SAP projects.”

Backups are stored directly on the Data Domain systems and tape has now been purged for daily and weekly backups at these sites. Once a month, a backup is made to tape at each of these locations and stored for DR purposes. Seven to 10 days of Exchange backups are stored to the Data Domain systems, and up to 30 days of database and SAP application backups are now also retained on the disk in order to shorten recovery time objectives.

In addition, backup administration is much easier. No longer does Vanden Bosch have to spend fifty percent of his week fixing tape units, driving to each location around the city to resolve local backup issues, or trying to find the right tape to satisfy an ever growing number of requests for restores.

“We have been receiving more and more requests for information on city records from the media, lawyers and so on,” said Vanden Bosch. “Over the course of any week, I’ve gained a few hours per day back thanks to Data Domain. Moving away from tape has given me more time to satisfy our many restore requests and facilitate restore requests on the fly.”

The City of Houston ITD is also happy with the service and support received from Data Domain. Vanden Bosch tells the story of a hard drive error that occurred during the middle of the night. This fault condition automatically generated an email to Data Domain support. When he woke up, he had already received a message telling him that a replacement drive would arrive that day.

Following its initial success, the city plans to add several more Data Domain DD690 systems in the near future which will provide additional capacity, performance and scalability. This is part of a broader centralization and replication initiative by ITD that includes Data Domain OST software with Veritas NetBackup. The plan is to conduct optimized replication of all data at its local sites to a central site all under direct control of NetBackup, and then perform monthly archiving to tape from that central system. Currently, it takes about a week of every month for the ITD staff to administrate and pull data from each site for monthly archiving. Once implemented, this time should be cut down to about one day per month or less.

“The goal is to eliminate tape for all but one centralized monthly archive,” said Nicholson. “We are well on the way to achieving this using Data Domain deduplication and replication technologies managed by OST and NetBackup. This will also eliminate many potential points of failure from our data protection process.”