you're reading...
Conferences, General

Are we really in control of data quality ?

In the past week the annual nordic IM conference IM2011 was held in Copenhagen with more than 200 participants, sponsors and both Danish and foreign speakers.

One of the topics discussed was the reason why large BI projects typically tend to be especially challenging or even fail. Data quality and data providing were unsurprisingly highlighted as possible answers.

Interestingly there was a significant difference in how this claim was regarded among the major BI software vendors on one side and on the other the BI customers who constituted the majority at the IM2011 conference.

Asked by a simple vote, the majority of the conference participants agreed that data quality and data providing remains a major challenge in BI projects, while four out of five suppliers accordingly expressed that they do not see that data quality typically pose a greater challenge. Instead the vendors outlined the most pressing challenge to be corporate use of BI in existing business processes – and that the real challenge is thus associated with business understanding and knowledge of the business users and the processes that BI solutions are supposed to support.

I myself am on a daily basis in dialogue with many different BI customers, and I am thus continually witness to some of the struggles and challenges that typically characterize large BI projects. For the same reason I can with some confidence, note that I agree with both statements. However do not mistake! – Data quality is definitely still a major challenge in many BI projects – although most BI customers through the years have worked intensely to try to improve it while ETL/ DQM tools as well as governance processes have been upgraded continuously.

Still a challenge!

Despite this, the situation remains that business users and analysts, BI/DW experts and consultants typically have a daily recurring challenge of trying to find out why sales volumes, turnover, vacation figures or the profit and loss statement do not align with the original transaction figures in the operational systems. There is however typically a common recurring cause of this.

The thing is that data quality in isolated systems typically is quite good – especially when it comes to transactional data. So if BI project’s primary task is to obtain these data and make it available for reporting and analysis as they are – then the challenge relating to data quality is certainly not as severe as previously seen. The problem is that this is not a realistic picture of how BI projects typically are scoped. Very few BI projects include scenarios where data is derived entirely from a single system, where data are not supposed not be enriched with business rules or where there isn’t a tight coupling between transactional and master data entities. And this is where the large BI projects most significant challenges arise.

Business needs and requirements are not simple – they are often complex and above all there is a very tight correlation between data entities across physical systems. This is the classic challenge for BI project – that is to control how data relate to each other across systems and business units to meet the necessary business requirements. Oddly this also confirms the argument that the real challenge has to do with corporate business alignment and convergence of the BI project deliverables. This is one aspect that really should be put on the agenda now that BI begins to make it’s way into enterprise business processes, thus resulting in so-called analytics or compositions.

Where are we then?

So all in all – are data quality and providing still one of the main reasons why BI projects fail or are extraordenary challenging ?. In a simple world, the answer is No – but it’s probably very few CIO’s or BI program managers who can argue that their BI initiatives exist within a simple and well defined scope. Therefore, data quality, data adoption and convergence are definitely still a challenge in many BI projects – typically due to poor master data as well as administration and understanding of business requirements related to the appliance of business logic and data transformation rules.

This consideration has a direct impact on how we prospectively could utilize and integrate BI into existing business processes. It makes it more important than ever that BI professionals understand and can relate BI to specific business processes – such as logistics, finance, controlling, HR, procurement, etc.

This is a requirement if the large BI projects should move into the BI 2.0 arena and we can really say that data quality is no longer a challenge. However it also requires a continued recognition that in a complex  enviroment there is still the need to focus on specific tasks, related to the quality and providing of data.


About Mads Frank

Mads is director within Platon Denmark and member of the danish management board . With 15 years of experience within SAP information management consulting his primary focus is on implementation and general management of information management projects related to SAP and SAP BusinessObjects - as well as general information management advisory. Mads’ primary professional competences is centered within the traditional data warehouse deciplines as well as general project management and methodology development. Mads further more specializes within the integration of third party ETL and front-end technologies with SAP - including Microsoft, QlikView, Theobald, Business Objects, etc.


No comments yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

%d bloggers like this: