Parallelize CHICKEN core tests

[siiky]

What?

This ticket is meant to be the place to discuss everything related to the parallelization of the (current) CHICKEN core tests (make check). It's not a "please do this" ticket! I'm willing to work on it.

I'd like to know if there's any interest in it, so please share your opinions, thoughts, and ideas!

Why?

The tests take the majority of the time of the whole build process. On my desktop:

• Build: ~3m32s (1 CPU); ~2m25s (2 CPUs); ~2m12s (3 CPUs); ~2m8s (4 CPUs)
• Test: ~7m38s
• Install: ~2m19s

On my RBPi2 the whole process takes hours -- I usually leave it running over night. On my RBPi4 it takes around 45min, which is not so bad, I'm still impressed with it, but "I want it all, and I want it now"! This is actually my biggest incentive, because of the salmonella tests.

megane said on IRC that there's been some talk about rewriting the tests in Scheme, but that it's not for the near future, which I can understand. However, I think parallelizing the current tests should be pretty maneagable, and quicker than rewritting them in Scheme (a guess before actually putting my hands to work, of course), so we could take advantage of those in the meantime.

How?

Assuming that there's interest, I'd like to know what others think -- ideas on how to implement, and how not to implement the tests. Related, I'd also like to know what's OK, and what isn't -- file structure, file names, number of new files in the repository, files created by the tests, etc.

Now, my current idea/plan: Keep everything contained in the tests/ directory, as it is now. Have a script for each test -- one for "repository search path", one for "types.db consistency", etc. Have an "environment" script that sets up the environment variables (CHICKEN, CHICKEN_PROFILE, ...), commands (compile, compile_r, ...), etc, to be used by each of the previously mentioned scripts. Have a Make target for each test, so the actual parallelization is managed by Make. Each test script should return the usual 0 for OK, and !0 for !OK, so that Make works correctly.

The first obvious problem is the tests' output. Right now, since they're sequential, everything is printed to stdout, and all is fine. But running in parallel that's not an option. My first idea to solve this was to save the output of each test script to a test-specific output file.

And the last thing I'm wondering about is whether having a test-specific directory inside tests/ is worth it or not. This is mainly to unclutter the tests/ directory, due to the tests' output files.

That's all I have for now! Hopefully I didn't miss anything stupidly obvious...