Preventing a recurrence of our present economic crisis requires:
High Quality Data
Effective analytic models
Business judgment to take action on findings
Every one of these objectives requires:
Business leadership
Process improvements
Increased compliance tracking
Much has been and will be written about the causes of the downfall of the economy and how to prevent a recurrence. Those of us who make a living managing or consulting on data management will use this opportunity to promote better data management and data quality plans. More timely, accurate and complete data about financial institutions, borrowers, mortgage lenders, securities and asset ratings would certainly have accelerated the knowledge investors, regulators and companies needed to change course.
While this article argues for establishing a robust data quality plan with a comprehensive data management program, this step alone will not prevent a recurrence of our economic woes. More needs to be done to improve the confidence in and transparency of the analytic models used by companies to forecast risk, trends and operations. Similarly, developing a business leader’s confidence to take the appropriate management action with the model's results is also needed.
If any of these steps are overlooked, then we have not learned our lessons and are doomed to repeat our mistakes. Let’s discuss the improvements needed in each step of the data continuum.
High Quality Data
Managing all business data to a 100% data quality level is neither realistic, nor an economical business goal. Therefore, business decisions must be made as to what data will be managed and what is “good enough” data quality. These decisions need to be made by business leaders using the data with support from the IT organizations and data experts in the company.
Critical business data that materially affects the company’s analytic models, regulatory reporting and critical business processes should be the first priority for data management. At the enterprise level, this analysis usually results in finding only 25-30% of the data collected across by the company is deemed critical.
After critical data is identified, comes defining data quality needs. The definition of “good enough” for critical data quality depends on the usage. For instance, data quality requirements for financial reporting will be different than data quality for modeling and analytics. Data users need to be asked to identify the data accuracy, completeness and timeliness dimensions required of the critical data they use to ensure their results are materially sound.
Once the data quality requirements from each data user are known then plans to measure and continually improve the critical data follow. Data will decay over time due to normal business activity. For example, customer data will decay at 4-7% a month. In economic downturns like today, that rate can be even greater. Think about your own personal contacts and how often the phones, emails or titles change. That is why an ongoing data quality program with an ongoing investment and dedicated team is required.
This is rarely the case, however. In a recent survey conducted by Information Difference, almost 2/3 of businesses reported they do not measure or monitor the quality of their data. Despite Sarbanes-Oxley and other regulatory pressures, data quality programs have not caught on as a means of validating adequate controls. Yet understanding the quality of the data with concrete metrics is fundamental to assessing the risks of the analytic models, not to mention the other critical business processes. It may take regulation to make a data quality program a mandatory compliance item in order to raise the priority of the program. A data quality program with metrics will provide much needed transparency and ultimately restore the confidence of the data used in key critical process.
It is equally important to understand and manage the external data used across the company. Most external data providers do not provide data quality metrics and are not motivated to do so for fear of devaluing their service. This needs to change in order to bring more trust and confidence to external data. Case in point, the lack of transparency in the AAA bond and securities ratings which turned out to contain high risk assets. [Who was responsible for making sure that the rating was understood and accurate?] Understanding the accuracy rating and data quality plans behind those ratings would have invited inspection earlier.
Lastly, too often these data quality activities are managed by the IT organization with little business engagement. Business leaders have to be more involved and ultimately drive the programs themselves with IT providing support.
Effective Analytic Models
Significant changes to our analytical models and our modeling organizations are necessary for moving forward. Many modeling organizations will tell you they saw some of these economic issues coming. The models were showing customer, market and credit risk was increasing and action was needed to mitigate risks. Unfortunately the modeling function was not convincing enough for business leaders to take actions. The credibility, capability and importance of the analytics and modeling organizations and their internal operations need improvements. Having high quality data for modeling is only one of the improvements required.
Analytic models take as input internal & external historical and current data, apply business rules, assumptions and mathematical formulas to yield results and recommendations. Modeling organizations are typically mathematical experts and their models are considered “black boxes” by most business people. Business people have little motivation to understand how these models work. Understanding how the models work involves understanding statistics and advanced mathematics, a skill which is eroding in U.S. colleges and universities. That which we do not understand we do not trust completely and this is true for analytic models. Because of a lack of trust, taking bold actions with modeling results is questioned and has to be constantly justified. Business leaders must take the time to understand how the models works by reviewing the business assumptions used in the model, the business rules used to process the data and the quality of the input data being used.
Modelers are typically not data experts and certainly not data management experts. They rely on the existing data experts in the company as well as the transactional systems and data warehouses to give them the data to run their models. Many times the same date is procured from different sources at different times. This creates differences in data throughout the systems. Missing data is imputed, eliminated or extrapolated with potentially risky consequences. As the economy deteriorates, the timeliness of data becomes more important but also less reliable. Even two week old data is useless in predicting the future of an ever depreciating asset market.
Modeling organizations are the loudest voice for data quality in most companies. The data demands of the modeling organization are great yet often times the modeling organization feels they are at the “mercy” of the data they get rather than as power brokers to get the data they need. Because they are not data management experts, oftentimes they don’t know how or what to ask to change to give them what they need.
This has to change. Future modeling organizations should include a data operations team whose job it is to procure the data from the same place for all the models, from a trusted source data warehouse, and to drive their data requirements to the source systems. They should be empowered by the company to be a champion in changing processes and systems to produce the data they need since this ultimately will lead to better analytics and decisions.
Business Judgment To Take Action On Findings
With the recommended changes made to ensure high quality data and models, the analytics will be more reliable, accurate and timely. Now comes the really hard change - acting on the findings. As we have said, many modeling organizations will tell you they saw some of these economic issues coming but were not effective in convincing business leaders. Ultimately, the business leaders still have to make a judgment call with the analysis. Either believe the models and make changes or stay the course. Too many chose to stay the course. They stayed because of competitive or economic pressures, lack of confidence in the models or just plain greed. Preventing this lack of action, despite proven high quality data and reliable analytics requires more management checks and balances to ensure these decisions are made collectively.
Business leaders would also benefit from management courses in data management and analytics. These classes are offered by industry groups such as Gartner and TDWI. The first time business leaders are introduced to analytics and data management is usually “on the job”. More can be done in the under graduate and graduate level universities to make data management & analytics a part of the business curriculum.
Finally, once neither the data nor the models can be used as an excuse for inaction, it comes down to a judgment call for which executives have to be held accountable. That becomes the job of senior management and the board of directors.
Summary
Whether by increased regulation, compliance tracking or management development, the role of high quality data for modeling and reliable analytics has to be prioritized to restore confidence in decisions made by business leaders. Business leaders must trust their data and models or invest in programs to increase that confidence so they can take actions confidently when the results are not popular or in line with current market trends.
