Future State of Data Survey

Common Ground

Our advice to the new Administration? Before launching a data-improvement initiative, check the basement and make sure the foundation is solid enough to support the transformation.

Improving even one of these important aspects of data management is a herculean task, but it becomes impossible if the underlying infrastructure can't support the solution. For some of the issues we surveyed, the connection back to infrastructure is straightforward. Storage issues and security must be addressed at the physical level. Cost and scalability have clear physical corollaries as well. But we tend to think of data quality, operational data access, and mixed workload as things to be worked through at the logical level or in surrounding business processes. This approach works in some environments, but there is no such thing as a purely logical solution in the world of large data, where many of us now live.

To facilitate data integration, you need a physical platform capable of handling twice or ten times as much data. Performing data quality routines against 100 billion rows of data requires heavy lifting. More complex scenarios where large numbers of users are accessing data in many different ways—through dashboards, reporting, ad-hoc exploration, and analytics—simply cannot be done without massively parallel processing and its physical counterpart.

Dataupia asked attendees at TDWI World Conference in Las Vegas, NV (February, 2009) their thoughts on what aspects the new U.S. Administration should include in its technology initiatives.

The collective advice from 134 surveyed BI and data warehouse experts to President Obama is unambiguous. The experts represented by the TDWI sample would urge the nation's Chief Data Steward to focus on Data Quality and Data Protection/Privacy while being mindful of the significant challenge posed by integrating data sources.

Here are the survey's questions and the results:

What will be the greatest data-related challenge for the new Administration?

What is the most significant feature that the Administration should consider in relation to any changes that may be made to its data infrastructure?

If you could encourage one data warehousing and/or business intelligence policy, what should the US CTO focus on?