Talk to your Tests - Text-Based Data Generation and Verification

A couple of years ago I worked through two books on creating games with Rust. One of them was Hands-On Rust.

After working through the book, I played around with Bevy and created a small 2D game. It was a fun little side project, and I did learn a lot about both Rust, and some basics of game development.

The most complex part of the game was the automated level generation algorithm I developed: I created a random world map, where each tile led to an auto-generated dungeon with several levels and different room designs.

The algorithm grew in complexity, to the point where verifying its correctness became difficult. I created the algorithm using TDD, so I had plenty of unit tests. Additionally, I wanted tests that created an entire dungeon, and checked whether it was traversable from start to end. (A path-finding algorithm helped with that.)

Now, the tests were green, but I had no idea whether the dungeons actually looked good. So I printed the generated dungeons to the console as the tests ran. If you ever try to print a 3-dimensional dungeon consisting of dozens of 2-dimensional rooms, you will quickly realize that this is not as easy as it sounds.

Eventually, the output was written to files instead of the console. Also, I added functionality to read the maps back in from those text files.

Dungeon layouts, which otherwise consisted of large amounts of objects, were now visible in text files rendered as ASCII art:

###############
#.OOOO........#
#......H..H...|
#.............#
#######_#######
      #.#
 ######_######
 #........XX.#
 #........XX.#
 ######_######

I re-used the idea of human-digestible representation in other projects since. It really is a very simple concept, and plays into the theme of writing code for humans, not machines.

Many developers write super-readable code, some even write super-readable test code. But often the data that is used to set up tests and verify results is cryptic at best.

For simple use cases, not a big deal.

When scenarios get more complex, however, it can become next to impossible to follow what the code is actually doing.

Consider, for example, a piece of code calculating prices for print orders. This is a requirement from a real-world project I worked on a while ago:

When creating offers for print orders, the system needs to calculate a price based on:

• The number of pages per print
• The number of copies
• The type of paper
• The machine times for: printing, cutting, punching, etc.
• Minutes of manual work per type of work (machine supervision, manual cutting, etc.)
• Calibration times and material per machine (different for each machine)
• Discounts based on number of copies
• Discounts based on existing customer contracts and negotiations
• Packaging costs: material and time or fixed costs as negotiated
• etc.

The calculated price was displayed in the price calculator's UI.

When writing the algorithm, I faced a similar challenge as with the dungeon generator: How can I verify that the calculated price is actually correct (not wanting to rely solely on automated tests)? At first I looked up the individual data points and did the calculation in an Excel sheet. That did not work for long.

So I added a little (i)-icon next to the calculated price in the UI. Clicking on it opened a pop-over with a very simple bootstrap table, showing each calculation step with minutes, sheets of paper, ink use, etc. I solely added this feature for my tests, intending to remove it later.

Once the application was used by my colleagues in the print shop, the questions started coming:

• "Why is the price so high, when I print on this machine?"
• "Why does it say that I need 1010 sheets of paper instead of 1000?"

I was able to answer those questions with a look at the breakdown table. Then I showed them how they can look the price calculation up themselves.

And the next feature requests came:

• Can we style the table so it is more readable?
• Can we add more details, like name of paper, etc.?
• Can we add information on whether the packaging costs are fixed or calculated?

What I intended as a temporary feature for testing became a prominent and useful functionality of the final product.

A last example from a recent project: I was working on a software product that allowed customers to plan shift assignments. The central calculation algorithm spit out a "calendar" for each department, showing which employee worked where and when.

Over time this algorithm grew in complexity. We already had a great way to display the final results: our custom-built calendar UI.

But the data fed into the algorithm was hard to set up. Not only did you have to provide information about an employee's shift and department, but also:

• Breaks
• Absences
• Holidays
• Shifts worked in multiple departments
• Shifts spanning 2 days (night shifts)
• Special rotations and overtimes
• ...

When writing a realistic integration-level test for an entire organization, you first had to set up an entire database worth of objects. By the time you set up the 5th employee in the 3rd department with their own unique night shift plan, special rotations, time tracking data, and their doctor's appointment on Wednesday, you are about ready to throw your computer out the window.

The solution was a data generation tool, that took text-based input and generated the necessary data. I make this sound a lot simpler than it actually was.

The data generator became a sizeable part of the code base. We not only used it for all higher-level tests and for manual testing.

But also to set up local development environments with different scenarios. And then we added endpoints to upload text files to seed test and demo stages. This was a much-used feature for presentation and demo sessions with potential customers.

Simply adjust and upload the file to the demo stage, wait for a minute, and voilà: you have an environment custom-made for the demo tomorrow.

Beyond Text

In some cases, I used visualizations, where text representation was still too complex.

As an example, I once wrote a piece of code that recognized material flow between departments by analyzing imported data from each department's source system.

It was simple to add a graphing library to the automated tests and create weighted nodes and edges by identifying recurring IDs in the imported data.

That allowed me to verify visually whether the algorithm recognized the flows correctly.

The theme is the same: make it easy for humans to understand what is going on in your tests.

Conclusion

The idea of text-based input and output for tests is simple, but powerful.

As your tests grow in complexity, any support for data setup, execution, and verification helps tremendously.

It makes debugging test failures and production issues easier.