In most organizations, there are custom reports designed in ERP to satisfy the reporting needs of business. Often, these custom reports are very similar to the standard out-of-the-box report with minor tweaks as per the business need. These tweaks can be the addition of a few extra columns, formatting changes, sorting changes, removal of unnecessary columns, and some additional calculations required for analysis and decision making. Sometimes, these tweaks are performed using the standard report design / logic.
When an upgrade is performed, custom reports need to be carefully tested for remediation and changes. For custom report testing, the below six-point strategy should be followed:
Testing of reports for data before upgrade - To perform the testing of reports for data before the upgrade, one test instance of the existing production application should be built using the same data backup as of the upgraded test instance. Now the reports should be executed in the upgraded test instance (with remediated code) and the test instance of the existing version (with existing production code version). Both the report outputs should be matched for same parameters. If the report outputs do not match, then that indicates an issue with the extraction of data before the upgrade. If the report outputs match, then it signals that the remediated code can provide reports as before. It should be ensured that this testing is performed before any data entry in both instances (i.e. upgraded instance and existing version instance). This will ensure that data in both instances are same and if there is any mismatch in report output, then it is an issue which needs to be fixed.
Testing of reports for data after upgrade - To perform the testing of reports for data after the upgrade, new transactions should be entered in the upgraded test instance. After entering new transactions, the report should be executed, and its output should be thoroughly reviewed. It should be ensured that the report output is tested for all possible test cases and not only just data entry, for example, in a custom report related to payable invoice, the report output should be tested not only for a standard, but various types of invoices and statuses. For the post-upgrade data, testing should be planned along with transaction entry testing so that data created in transaction entry testing can be re-used for report testing.
Report format - Post-upgrade report format should be verified with pre-upgrade report format. This verification will ensure that there is no formatting issue due to upgrades / changes / code remediation. This testing should be performed along with the testing of data before upgrade. With this approach, efforts for format testing will be reduced.
Testing of data which was partially processed before upgrade - Report output should be tested with the processing of transactions which were initiated before upgrade. For example, in a custom report of payable trial balance, report output should be tested with those invoices which were created before the upgrade but are paid after the upgrade. This testing will ensure that partially processed data is shown correctly in reports.
Report parameter - Report parameters should be tested thoroughly. Many times standard value set are used for a list of values in report parameters. When the upgrade is performed, then these values may get modified in new release. Also, when there is a custom value set which is table-based and if the table is obsolete in upgraded version, then that value set needs to be remediated. For any such items, the parameters of the report should be thoroughly tested.
Report execution after go-live - A lot of reports are part of the standard period close package. These reports are executed only during and after the period close. After the upgrade, these reports should be executed before approaching period end. Alternatively, a mock period close is suggested to identify issues before approaching period end. This will ensure that if there is any issue in reports, then it is identified a lot early before approaching period end and can be fixed. With this, the period close timelines will not be affected adversely.
In summary, it can be said that a good test strategy should contain detail section / plan for report testing to make the upgrade project successful.