In the past years the focus on costs in all business processes have increased with the objective to continuously reduce cost and optimize processes to stay competitive.
Does this impact the BI and Data Warehouse development processes as well? The answer to this is obvious – all projects are constantly under pressure to run faster and complete earlier with no additional costs. Is this possible? To some extend I would answer yes and allow me to explain. In Platon we continuously capture best practice from our experiences with IM projects (BI, Data Warehouse, MDM and Data Governance etc.) and integrate this as guidelines and templates in our knowledge framework, Platon Insight® which means all consultants have immediate access to the complete sum of best practices collected and developed over the past 12 years. To benefit even more on this knowledge we build accelerators for various means with the objective to speed up BI and DWH activities without compromising quality or costs. The Data Warehouse Framework (DWF) is an example of these accelerators. Over the years we have built numerous Data Warehouse solutions which all have a lot of ETL processes to extract, transform and load data for the BI solutions. Many ETL processes basically do the same, so we decided to standardize and automate as much of this as possible. With the DWF, the scripting of ETL processes is automatically generated once you have entered basic parameters of your measures and dimensions in the meta data layer of the DWF. The DWF does not completely eliminate the ETL developer but it takes care of the more trivial ETL activities leaving the more complex and value adding (and interesting) ETL activities to the ETL developer.
The DWF not only makes the ETL developers job more interesting it’s also a benefit to the BI/DWH department as it enforces standardization and quality. With standardization comes benefits as person independent solutions – all your ETL developers will much easier be able to understand a solution and ETL flow that was built by someone else which makes support and maintenance speedier and cheaper.
The DWF also makes it easier to go from project state to operations state. I’m sure many of you have had a challenge when you’ve finished the project and want to handover the project to operations so you can start the next project. Did some of the support and operational monitoring tasks stick to you even though you’re on the project development team?
The standardization within the DWF will also help the handover process as operations are standardized. The operations persons know the structure and need only to be told the specifics within each of the structural components of the solutions. Furthermore the DWF comes with standardized metadata and operations reporting so monitoring and maintenance is supported. All ETL flows are monitored the same way which makes it a lot easier to decide which actions are necessary to support a solution.
The metadata reporting is a great help for the DWH architect as it is very easy to get a complete overview of the entire DWH solution. This is important when new requests emerge and you need to decide if and what architectural or modeling changes are necessary. This accelerates your analysis phase as you quickly know if the DWH will support the new business requirements or if you have to enhance your current solutions with additional sources, measures or dimensions. The metadata reporting is also always up to date – you don’t have to rely (doubt) the static documentation on paper or in an electronic document – it’s right there at your fingertips.
So if you feel the pressure in your BI and DWH projects, start standardizing as much as possible without compromising agility, quality and costs. Even though your DWH project is already started it can make sense (or even save the day) to swithch to a standardized approach. Just, recently we were engaged in a DWH project not using a specific methodology or develpoment standard and we could see the risk of missing the deadline growing bigger every day – a decicion for using the DWF was made and we finished in time even though we had to transform the already developed project components to the DWF structure.