In the past week the annual nordic IM conference IM2011 was held in Copenhagen with more than 200 participants, sponsors and both Danish and foreign speakers.
One of the topics discussed was the reason why large BI projects typically tend to be especially challenging or even fail. Data quality and data providing were unsurprisingly highlighted as possible answers.
Interestingly there was a significant difference in how this claim was regarded among the major BI software vendors on one side and on the other the BI customers who constituted the majority at the IM2011 conference.
Asked by a simple vote, the majority of the conference participants agreed that data quality and data providing remains a major challenge in BI projects, while four out of five suppliers accordingly expressed that they do not see that data quality typically pose a greater challenge. Instead the vendors outlined the most pressing challenge to be corporate use of BI in existing business processes – and that the real challenge is thus associated with business understanding and knowledge of the business users and the processes that BI solutions are supposed to support.
I myself am on a daily basis in dialogue with many different BI customers, and I am thus continually witness to some of the struggles and challenges that typically characterize large BI projects. For the same reason I can with some confidence, note that I agree with both statements. However do not mistake! – Data quality is definitely still a major challenge in many BI projects – although most BI customers through the years have worked intensely to try to improve it while ETL/ DQM tools as well as governance processes have been upgraded continuously.
Still a challenge!
Despite this, the situation remains that business users and analysts, BI/DW experts and consultants typically have a daily recurring challenge of trying to find out why sales volumes, turnover, vacation figures or the profit and loss statement do not align with the original transaction figures in the operational systems. There is however typically a common recurring cause of this.
The thing is that data quality in isolated systems typically is quite good – especially when it comes to transactional data. So if BI project’s primary task is to obtain these data and make it available for reporting and analysis as they are – then the challenge relating to data quality is certainly not as severe as previously seen. The problem is that this is not a realistic picture of how BI projects typically are scoped. Very few BI projects include scenarios where data is derived entirely from a single system, where data are not supposed not be enriched with business rules or where there isn’t a tight coupling between transactional and master data entities. And this is where the large BI projects most significant challenges arise.
Business needs and requirements are not simple – they are often complex and above all there is a very tight correlation between data entities across physical systems. This is the classic challenge for BI project – that is to control how data relate to each other across systems and business units to meet the necessary business requirements. Oddly this also confirms the argument that the real challenge has to do with corporate business alignment and convergence of the BI project deliverables. This is one aspect that really should be put on the agenda now that BI begins to make it’s way into enterprise business processes, thus resulting in so-called analytics or compositions.
Where are we then?
So all in all – are data quality and providing still one of the main reasons why BI projects fail or are extraordenary challenging ?. In a simple world, the answer is No – but it’s probably very few CIO’s or BI program managers who can argue that their BI initiatives exist within a simple and well defined scope. Therefore, data quality, data adoption and convergence are definitely still a challenge in many BI projects – typically due to poor master data as well as administration and understanding of business requirements related to the appliance of business logic and data transformation rules.
This consideration has a direct impact on how we prospectively could utilize and integrate BI into existing business processes. It makes it more important than ever that BI professionals understand and can relate BI to specific business processes – such as logistics, finance, controlling, HR, procurement, etc.
This is a requirement if the large BI projects should move into the BI 2.0 arena and we can really say that data quality is no longer a challenge. However it also requires a continued recognition that in a complex enviroment there is still the need to focus on specific tasks, related to the quality and providing of data.