Visualization Driven Migrations, a Cobol Story (Part 1)

Originally, we had offered to build them an ERP system, but after a make-or-buy analysis the client (wisely) decided to buy an existing product. They liked what we had done earlier in a proof of concept to extract data from their system, so asked us to help them getting all the data migrated. I've had a terrible experience with a migration a few years earlier, so this time I pushed for a much better approach: driven by daily visualizations, and steering on value and risk. Enable fast decision making by the most senior stakeholder needed.

In the history of moldable development, this was a significant step for me, really demonstrating the value of building many tiny tools to aid decision making.

Complex vs complicated

Migrations are mostly a complex problem. They fail so often because they are treated as a complicated problem. The devil is in the detail, and it is mostly impossible to get sufficient expertise available to solve a migration problem as a complicated problem, where you put all the needed experts in a room together and they can come up with a plan that can then just be executed. It almost always works much better to create a very fast feedback loop, where you can run lots of tiny experiments to see what works and what does not.

In this case, not all of that was available. The one current maintainer had just come back to working again after two years sick leave after a heart attack, and the original developer had passed away a decade before. The incentives for doing a migration were good: the current maintainer wanted to retire. Also, the one available copy of the Cobol compiler used a printer port dongle for copy protection, and the only links on the internet about it talked about the company having been bought by MicroFocus a decade ago. No proper documentation was available.

We understand well how to do migrations, but we started with insufficient knowledge about this old system. This Cobol system uses files to store data, and uses different Cobol programs that are changing those data files. To avoid having to duplicate the data definitions all the time, the Cobol program files refer to included files.

We wanted to get a better feeling for what parts of the software might be complex, so we decided to make a quick visualization of these files. Here we show a grid of all the available Cobol programs as the large rectangles, with nested in them a square for each include file. They are colored and sorted based on their size, with the white ones tiny, green ones small, yellow ones a bit larger and grey and black ones even larger.

And the red ones are the missing ones. At first we had many more missing ones, but after a few phone calls with the current maintainer we still only had the results shown. 285 Cobol programs, of which 15 refer to files that don't exist. How much time would we need to solve that problem? Inspecting and reading the code of those files would take us days.

Fast feedback to the rescue

We had been using the MOOSE software reengineering environment for this work, which supports creating visualizations very fast. This one only took a few dozen lines of code. To create a fast overall feedback loop for this project, we aimed at presenting a visualization to the project sponsor and overall decision maker every day, that would allow her to decide in 5 minutes what we should work on the next day. The COO of this mid-sized company is very busy, and it is difficult getting more of her time regularly. And she has both the knowledge, and the decision making power to ensure that we can have time from, and talk to the right experts to make good progress. Here, she called an expert into the meeting who could tell us in 5 minutes after looking at the names of the Cobol programs that they are not important, and that the ERP system supported more functionality than they used.

To be continued