"Business technology news for Europe's senior executives...."
New Account

The Magazine

Issue 12

We speak to the key decision-makers looking to steer their businesses through these choppy economic waters.

E-magazine
  • Previous Issues

Blog

Spencer Green
Chairman, GDS International

Sales and the 'Talent Magnet'

A lot is written about being a ‘Talent Magnet’, either as a company, or as President. It’s all good practice – listen, mentor, reward, provide clear goals and career maps. Good practice for the employer, but what about the employee?
25 May 2011

Eye-popping ROI on building and maintaining a data warehouse: a case study


L arge American insurance company saves more than 90 percent on deployment, eliminates maintenance costs, gets the world’s fastest performance and gives users 100 percent train-of-thought analysis.

This ROI story of a large American insurance company begins with a review of the history of data warehouse database platforms, as this provides the basis for their cost comparison and astounding decrease in the cost of owning and maintaining a data warehouse.


The data warehouse industry is long overdue for an overhaul. Business analytics grew from the need of organizations to mine their transaction systems for valuable information that could help savvy business analysts make more intelligent business decisions. Since relational database technology was already familiar to most organizations, they used this technology to house data integrated from multiple systems, and data warehouses were born. Key early relational data warehouse vendors were, and still are, IBM’s DB2, Oracle and Teradata. The insurance company in this story was a DB2 customer.

The problem is that relational database technology was developed for high-speed transaction processing of records, not for meeting the flexible access needs of business analysts. To address this, vendors, over the years, have come up with creative ways to make data repositories more flexible for users by creating purpose-built data marts. By making them application-dependent, database designers are able to better balance the trade-off between speed and flexibility when building a data mart. The insurance company is this story had a DB2 data mart in its customer risk department.

Unhappy with the costs of maintenance and inadequate query performance, the customer risk department began its search to replace its system. The search led them to newer column database vendors Sybase IQ and Vertica, and data warehouse appliance vendors Netezza and Teradata.

The Payoff Has to Make Sense
The company found that while column databases provide an interesting alternative to relational systems, they have issues with loading speed at high volumes and with performing other record-oriented processes, such as the exporting and reporting of full records. Column-based systems were faster and more flexible, but the ROI simply didn’t make sense. The costs of deployment were high, the ongoing maintenance would be about the same as their current system and query performance improvements were not significant enough to warrant the expense.

The company also reviewed the data warehouse appliance vendors. They offered phenomenal speed, but they have all the same problems, and worse they still use old relational technology, and the hardware is proprietary and very expensive.

The most significant drawback of all of these technologies is the effort required to build and maintain a relevant, useful data warehouse for business users. They simply fall short when a business process (information need) changes because it requires a time-consuming process of re-architecting, reprogramming, re-optimizing, recalculating and redeploying the data warehouse or data mart. This is not only expensive, the opportunity costs are huge.

In the insurance business (and for many other businesses in general) innovation and speed are critical. “Fast and flexible” became the mantra in the customer risk department, but their systems couldn’t keep up and it was costing them a fortune. Their business processes and information needs are fluid – changing and fine-tuned continuously. And the bottom line for them was that any purpose-built, fixed-structure analytics data mart, no matter how fast or cheap, was going to be a drag on productivity in this environment. They are expensive; time-consuming to build, deploy and maintain; and difficult to use for people who don’t think like data modelers.

Taking Control of Cost Factors
In late 2008, one of the company’s data warehouse architects learned of a new data warehouse platform –  correlation database technology from illuminate Solutions – that promised to solve many of the customer risk department’s cost issues.

The correlation database is a breakthrough in data storage models, a breakthrough in data warehouse deployment processes and a breakthrough in analysis flexibility. The benefits of the platform are quickly obvious to both large, technologically sophisticated organizations as well as savvy smaller organizations that have yet to attempt to deploy a data warehouse or advanced BI and analytics. It is a game-changing technology that could fundamentally, and permanently, change the world’s approach to data management.

Correlation database technology automatically designs and builds data warehouses using a data-generated structure that’s created during the load process, so there’s no upfront design work. The database’s storage technology is extremely compact and fast, and the unique data storage approach gives users 100 percent train-of-thought analysis flexibility. This technology eliminates the IT project backlogs and delays from user requests because there are no requests – all dimensions are always available for analysis, and integrating new data sources is as simple as loading the new information. The database does the integration work.

Cost of Deployment Drops by Over 90 Percent
OLD COST STRUCTURE: The insurance company’s IT department estimates that the cost of deploying an internal relational data mart is $4 million. The hardware and software cost about $3 million, and the labor to design, program, and deploy it is approximately another $1 million.

NEW COST STRUCTURE: Each data mart deployed using correlation database technology is estimated to cost less than one-tenth of the company’s relational technology. It projects the deployment of its first correlation data mart will cost approximately $300,000 for hardware, software and labor – an astonishing 92.5 percent reduction in deployment costs.

Cost of Maintenance Drops to Nearly Zero
OLD COST STRUCTURE: The insurance company’s IT department was responsible for maintaining the data mart in the customer risk department. Changes to accommodate new business requirements were charged at $250,000 for deleting a single table. In 2008, the cost to the department of deleting five tables was $1.25 million. The cost is $17,000 to make a DB2 modeling change.

NEW COST STRUCTURE: The correlation database has a flexible, not fixed, structure, so there are no tables to delete and no modeling changes. In addition, new data sources are automatically integrated. So there are, essentially, no maintenance costs.

How Is this Possible?
Without going into a great technical detail, central to its uniqueness and value is the speed of deployment of a correlation warehouse. The technology uses a unique process driven by a data-generated schema.

A database schema is what defines the structure of the data warehouse and, in correlation database technology, it’s automatically generated during the loading process. This is achieved by disassembling records and letting the data themselves create a sophisticated indexing system.

In addition to fast deployment, it’s simple to add and use data from nearly any source at any time. The correlation database does the work of integrating new data sources with its data-generated schema. No matter how many records are loaded, changed and deleted, the database always has an optimized physical structure.

Below is a representative timeline that illustrates where cost savings are gained during the deployment process. White papers on illuminate’s correlation database technology can be found on its website: www.illuminateinc.com .

Figure 3: Building a data warehouse with illuminate.

Cost of Lost Opportunities Can Only Be Estimated
Although the cost savings on deployment and maintenance are enough alone to justify the purchase of illuminate’s correlation data warehouse platform, the company is looking forward to realizing significant earnings as a result of the system’s providing its analysts with train-of-thought analysis capabilities.

Saving $120,000 and a delay of three to six months for each complex report when an analysis can’t be concluded is costing the company significant amounts in lost revenue. Although it’s still too soon to know how much the company can actually earn from having the information available on demand, the internal estimates are huge. As a point of reference, an illuminate insurance customer located in Europe estimates that their improved customer retention rate generates an astounding 30.000.000 Euros in additional revenue per year.

Train-of-Thought Analysis Is the Key
Today, to be a serious information analysis tool, a data warehouse must respond to people in a manner that’s flexible and intuitive – similar to the way people respond to each other. Without an iterative, or incremental, interaction, analytics systems are simply customized report writers and query systems.

The problem is all data warehouse technologies – other than correlation – can answer a question if, and only if, both the data structure and a query have been designed to answer the specific question. These systems are useful for answering “typical” questions and monitoring the routine activities of a business. However, they grind to a halt when an information requirement isn’t known and planned for in advance.

You don’t know what you don’t know. Many business users often face business problems that have no clear path to resolution. A correlation warehouse allows them to create and execute queries incrementally – using train-of-thought analysis – to quickly and easily explore different paths through the database until they arrive at the sought-after answers.

There is no need for a business user or other information analyst to “think like the database” and no need for IT to build or reprogram data cubes or write complex SQL queries. And most importantly, there is no need for people to limit their questions to the “typical,” backward-looking, routine-activity-monitoring queries common to current information infrastructures.

Conclusion – Correlation Database Technology Fundamentally Changes the Business

While other alternatives to relational technology – column databases and data warehouse appliances – offered the insurance company incremental improvements in speed and flexibility, correlation technology truly changed the game. Because a correlation database builds its own comprehensive index on the fly during data loading, there is no expensive predesign process required. Initial deployment costs are an astonishing 92.5 percent less than with a relational system. And since new data sources can be integrated automatically using the correlation database’s unique data-generated schema, there is never a need to restructure the data warehouse. Maintenance costs are virtually eliminated.

Even more significant than the cost savings, however, is the unprecedented level of analytical flexibility the insurance company’s customer risk department now has. It can now not only answer questions faster but, with virtually instantaneous train-of-thought analytical capability, it can answer questions to which competitors simply can’t respond. Correlation database technology changes the game – for this company’s customer risk department and the data analytics world.