There is, and I believe there always will be, a gap in what large Business Intelligence (BI) and reporting software systems can provide to serve the needs of business professionals.
BI/Reporting software has come a long way since the 90s when I started down the BI/Reporting path. But despite the great advances, building a solution using Tableau or Qlik, or other options, is very expensive. Even if a company invests and builds out a BI solution, there is still a gap, where certain managers or analysts will not be comfortable using the “self-serve” BI solution, or even if they are, it will not have the data needed to analyze.
Building a BI solution will at best fall into the 80/20 rule, where the solution will not be able to help at least 20% of the demand. From my experience a more realistic goal will be more in the 50/50 range, depending on the nature of the business and team. Much of the need is often not for self-serve analysis, but complex data sets that have been processed from multiple sources, and then scheduled on a regular basis.
Every management team I’ve seen jumps on the idea of a silver bullet where the sexy BI solution will fulfill all the needs of senior management, program and product management, and analysts. Every time I’ve seen it fall short of expectations.
Without intending it, I’ve made a career out of building high-end reporting/BI solutions to fill the gap for those who need reports that will help them effectively get their job done. The “white glove” solution are custom Excel reports I build using VBA. I have built a program that pulls data from a data warehouse, which may be processed on server-side stored procedures, into Excel and then format it. It can have multiple sheets, graphs, a data dictionary sheet and an overview sheet. Data can be properly formatted (e.g. numeric IDs as text not to drop leading zeros), the headers fixed, filters on, colorized, etc.
In the end, almost all managers/directors and analysts are comfortable with using Excel, so unlike the BI solutions, there is no training required for the recipients. Most of the time, these professionals can do quite sophisticated processes such as vlookups and pivot tables. This allows them to ask for a data set, maybe even integrate it with their own data, and then transform it into a format that they can slice and dice it to analyze the data and make key decisions to get their job done. This works for both operational and strategic reports. In the end, this may be a client list that needs to be analyzed to find an upgrade load list (operational), or to create metrics to identify top performers (strategic).
Filling the BI GAP requires seasoned professionals with strong SQL skills that also understand the business requirements. They need people skills as well as technical skills. They work in-between business and technology. More times than not, the “ask” for data isn’t what is really need. A process is needed to step through what questions they want answered and what they are going to do with the data.
I have yet to find such a job description on LinkedIn, but the role is critical in larger organizations. Job positions focus on on programmers and dbas on the tech side to build the data warehouse and BI solution, or the business as data scientists working with the BI tool. But most situations do not require a statistical analysis, rather a deep understanding of the product and ability to analyze and process the data to provide the deliverable.
Microsoft SSRS is the closest solution that provides Excel reports based on Stored Procedures, and that is a good solution to “push” reports out on a schedule. It doesn’t create as elegant a report as customized Excel reports, but it works great for operational reports that are list-oriented, and works great for scheduling large numbers of reports. It is the scalable solution for the white-glove customized reports, and is needed if the GAP is sizable.
There will always be a GAP between what a self-serve BI system provides and what business professionals need from a data warehouse. Requirements are always popping up new and then changing. In today’s Agile environment, timelines are crunched even tighter, and a self-serve system can’t keep up with the shifting sands of business requirements, or sudden “P1” tickets that have a client impact if not analyzed and resolved immediately.
A BI system and a white-glove “pushed” reporting system should be seen as complementary and part of a holistic approach to providing business intelligence to your team.