System Crash

Jones Regional Medical Center is a large academic health center. With nine hundred beds, Jones had forty-seven thousand admissions last year. Jones frequently has occupancy in excess of 100 percent, requiring diversion of ambulances. In addition, Jones had 1,300,000 ambulatory and emergency room visits in the past three years.

Jones is internationally renowned for its research and teaching programs. The IT staff members at Jones are highly regarded. They support more than three hundred applications and twelve thousand workstations.

The admitting system at Jones is provided by the vendor Technology Med (TechMed). The TechMed system supports the master patient index; registration; inpatient charge and payment entry; medical records abstracting and coding; hospital billing and patient accounting; reporting; and admission, discharge, and transfer capabilities.

The TechMed system was implemented twelve years ago and uses now-obsolete technology, including a rudimentary database management system. The organization is concerned about the fragility of the application and has begun plans to replace the TechMed system two years from now.

INFORMATION SYSTEMS CHALLENGE
On December 20, the link between the main data center (where the TechMed servers were housed) and the disaster recovery center was taken down to conduct performance testing.

On December 21, power was lost to the disaster recovery center, but emergency power was instantly put in place. However, as a precaution, a backup of the TechMed database was performed.

During the afternoon of December 21, the TechMed system became sluggish and then unresponsive. Database corruption was discovered. The backup performed earlier in the day was also corrupt. The link to the disaster recovery data center had not been restored following the performance testing.

Because there was no viable backup copy of the database, the Jones IT and hospital staff members began the arduous process of a full database recovery from journaled transactions. This process was completed the evening of December 22.

The loss of the TechMed system for more than thirty-six hours and the failure during that time of registration transactions to update patient care and ancillary department systems resulted in a wide variety of operational problems. The patient census had to be maintained manually. Reports of results were delayed. Paper orders were needed for patients who were admitted on December 21 and 22. Charge collection lagged.

Once the TechMed system was restored, additional hospital staff members were brought in to enter, into multiple systems, the data that had been manually captured during the outage. By December 25, normal hospital operations were restored. No patient care incidents are believed to have resulted.

DISCUSSION QUESTIONS

1. If you were the CIO of Jones Regional Medical Center during this system failure, what steps would you take during the outage? What steps would you take after the outage to reduce the likelihood of a reoccurrence of this problem?
2. The root cause analysis of the outage showed that process, technology, and staffing factors all contributed to the problem. What are some of the likely factors? Which of these factors do you believe are likely to have been the most important?
3. If you were a member of the audit committee of the Jones board of trustees, what questions would you ask the CIO?
4. What issues and problems should a disaster recovery plan prepare for? How does an organization determine how much to spend to reduce the occurrence and severity of such episodes?