An Example of the Pain Associated with an IT Upgrade

Last blog entry I discussed why IT upgrades or system changes often result in pain. This time I will give an example to illustrate the common issues that companies face when they attempt to upgrade a system.

In this example, the vendor’s legacy system provides tabular reports in flat text files and XML based data extracts. This worked well given the structure of the system’s database (mainframe based) until the volume processed by the system grew dramatically–a reflection of the customer’s growing business. The customer made a contractual commitment to upgrade to the next generation system provided by the same vendor which is capable, foremost, of handling the larger volume.

The newer system’s ability to process larger volumes of transactions is based on a three tier client-server architecture with a standard relational database management system and application servers in the middle tier. The new database was designed to handle large volumes of data with very low latency. As such, the database design is optimized for transaction processing.

To support reporting, the system replicates transactional data to an online data store (ODS) which is essentially a copy of the transactional database. To improve reporting response times for a select number of the reports, the vendor added a data mart which pre-aggregates some of the data that requires substantial processing prior to display in reports. The reporting system pulls data from the ODS and data mart and displays the results in a web browser interface. The developers that produced the reports took advantage of the new more flexible browser interface and designed many of the reports so that their structure was no longer tabular. For example, many reports in the new system display summary rows amongst detail rows and allows users to collapse the report to display only summary rows, or open select details.

To replace the data extracts from the legacy system, the vendor offers customers the ability to query the ODS directly on the new system. Because the ODS is hosted on an industry standard database system, the opportunities to extract and load the system’s data into other systems are much greater than with the XML extracts on the legacy system. To take advantage of these changes, however, the customer must change their approach to extracting data from the system. Some customers embrace such change. Others prefer that the new system maintain strict compatibility with the legacy system, perhaps even at the cost of performance.

One customer expected that they would obtain the best of both worlds: Dramatically improved performance and 100% (or very near) compatibility.

The resulting pain is based on their somewhat unrealistic expectation. This expectation may have resulted from invalid assumptions on the part of the customer, misleading impressions from the vendor, or more likely, both.

In particular, the issue arose when the customer realized that the new reports produced output differently than the legacy system and thus all of their internal processes that pulled information from reports and loaded the data to other systems would not longer work. Rather than investigate the potential benefits of the ODS, the customer opted to rely on contractual terms in an effort to force the vendor to enhance the functionality of the new system. As you might imagine, this approach proved time-consuming and painful.