If a company’s software migration is anything larger than tiny, it will require a significant amount of reporting and Business Intelligence (BI) analysis. Management rarely understands what is needed to properly support the data needs for a large migration project. Two common mistakes will be either to underestimate the data requirements and request that existing staff manage the project by spreadsheet, or they may not underestimate the scale but rather fall into the trap of buying/using expensive corporate level BI software tools that are great for many things for data analysis, but not very good at providing the types of reports a migration needs.
Analyzing and reporting by spreadsheet just can’t work because the data is a moving target. Clients and products and entitlements are constantly changing. A list will be correct for a given day, but the following week it will have changed. Keeping on top of the delta is a nightmare if done by spreadsheet. The data is also being cleaned and transformed, called the “data alignment” process, so that an automated program can move all the data into the new system without errors. This needs to be constantly checked until JUST before the migration event when all client entitlements get frozen and the migration happens, usually during a weekend when users are locked out of the system until the migration is complete (hopefully quickly).
The other way data support gets misaligned when management decides to spend TOO MUCH money on corporate level BI tools. These tools are powerful, but are really designed to build in a slowly changing, stable, environment and then giving the ability for analysts to slice and dice the data. This is NOT what a migration needs. It needs the ability to build complicated reports quickly that can be “productionalized” and replicated, and pushed out the the team as highly formatted, pre-digested, spreadsheets. They need to be sent over and over again.
For a large migration project, which can take years, tens of thousands of reports may be run and sent out. What are the characteristics of the system needed to support a large migration project? A few are as follows:
- Data Warehouse — For a large migration project, most likely there will be legacy systems and multiple data sources, including the new software, which need to be pulled into a common warehouse. Large migrations will occur in “waves” over years, and will have some of the clients on the old system and some one the new system. Data on both systems are needed.
- Agile — Migrations by their nature are not completely predictable, where all reporting needs are specified and built in advance. Critical issues will arise and need quick deep dives into the data and reports created. This may be browser incompatibilities, new compliance issues, product issues that arise after clients are migrated, etc. This can not be production environment with a 4 month turnaround. New data sources need to be brought into the warehouse, data analyzed, and reports developed quickly.
- Automated Reports — Many reports will need to be run on a monthly, weekly or even daily basis. The repetitive nature of the reporting requires it to be built for batch reporting.
- Heavy processing — Some of the analysis is very intense, and requires some serious database horse power. Most migration will need full information at the client, user, account, platform, billing levels to name a few. This is a lot of data with often comes from varying systems with complex relationships to tie them all together. Doing billing analysis, to make sure the clients are properly billed on the new system, may require modeling and a lot of processing. Many reports aren’t simple dumps of data but require extensive programming and processing.
- Highly formatted — In the end, almost all staff will want Excel spreadsheets. But when sending large lists with multiple sheets, these need to be highly stylized. Excel files will need multiple tabs/sheets, fixed headers, filtered, with nulls removed and numbers formatted as currency or with commas, and some numeric codes treated as text so they don’t lose leading zeros. The list goes on. This is too much from BI systems the export Excel files as a simple dump.
- Segmentation — Large migrations take a long time and can take years, with many “waves” of migration events, usually 2-4 times a year. To line of clients to migrate at the right time requires them to be segmented in many different categories, so they can be properly grouped and their communication, data alignment, and timing of their migration is done properly.
There are five categories of reporting needed to support a software migration:
- Management — High level reports giving summaries of the migration status and indicators. These are the top of the pyramid, and are essential to give perspective.
- Decision Support — Reports for business analysts and product support to facilitate key decisions and analysis. Recipients of these reports can be very diverse, from product folks, billing, sales, etc. These reports take a lot of experience and creativity to deliver data in a format where staff can make decisions.
- Data Alignment — Reports to operations staff that usually consist of lists with data that needs “alignment”, such as address cleansing, or other situations to support the data rules of the new software system.
- Communications — A software migration requires multiple communications with the clients. A key aspect of the migration is segmenting the clients as to how they use the product, and the communications will be tailored around each segment. This necessitates that the migration reporting system be used to cut lists to communicate with the clients. This is a critical part of the process, since senior management will be very unhappy if clients have a bad experience.
- Technology/Load Files — Any software migration will have automated tools that will move the client/account/entitlements/other data from the legacy system to the new system. A robust reporting system can be used to export the actual files that will be “locked and loaded” in the migration tool. This is really an Extract/Transform/Load (ETL) process which is the culmination of all the analysis and data alignment work.