Back in the day, BI vendors like Tableau, Qlik, and other tools from that vintage gained rapid traction in the corporate world by selling directly to the business as opposed to the traditional route of selling to IT. Business users from that time were starved for data and oftentimes very frustrated with their IT departments or in all out war with them over data access. These tools were relatively inexpensive and the vendors all promised immediate gratification without having to involve IT.
As a result, these tools were rapidly adopted and quickly started to create security, governance, and maintenance nightmares across organizations. Reports and dashboards popped up like mushrooms overnight, but these reports and dashboards cut a lot of corners and deliberately went against commonly accepted best practices in data management that had existed for over 35 years. In the zeal for “unlocking the data” and democratizing data access, organizations now struggled with an uncontrolled proliferation of reports with different reports providing different answers, without a “single version of the truth”. Complex data transformation logic would also be attempted inside these BI tools only to discover that the tools could not handle the data volumes or that maintaining this logic inside the BI tool was needlessly complicated and invited chaos when it came to support and maintenance.
In the years that followed, organizations had to reckon with the unintended consequences of their ungoverned BI programs. Proper data architecture, data governance, and adherence to data management best practices were back in vogue. Organizations realized that the BI layer was nothing more than the “last mile” in the data tool chain and that the reports and analytics were only going to be as good as the underlying data infrastructure powering the BI layer.
This was an expensive and time consuming journey for many organizations and we have all seen the career limiting moves of folks who championed a BI tool centric worldview. Sadly we still see this behavior and diversion from best practices which brings me to the title of this blog post.
One of the biggest signs of an immature data program is focusing on data access or BI reporting to the exclusion of having proper data architecture, governance, and data management tooling in place. Too often, the knee-jerk response to scaling or evolving an existing data program is to throw more BI analysts at the problem. This is not the right approach. Before you hire another BI Analyst, take time to ensure that your underlying data infrastructure has been properly set up and implemented, otherwise you will just create more ungoverned BI content that will lead to greater support overhead and dissatisfied users.
At Waterloo Data we are frequently called upon to perform a “health check” of existing data management platforms. The most common symptoms we see when evaluating legacy infrastructure is that reports are taking too long to process and run or they won’t run at all. In many cases, we also see a lack of consistency in common metrics across different reports, and when a metric is adjusted or recalculated, the changes don’t propagate to all the underlying reports. These are common symptoms that something isn’t right with the underlying data architecture and infrastructure powering your BI layer.
