Parallelise HelpCmd golden tests

[palas]

Changelog

- description: |
    Parallelised help golden test generation and validation
  type:
  - optimisation

Context

This is a proposal to parallelise help golden test generation. In my computer, it reduces the execution times from 27 seconds to 3.

How to trust this PR

Don't trust it

Checklist

• Commit sequence broadly makes sense and commits have useful messages
• New tests are added if needed and existing tests are updated. See Running tests for more details
• Self-reviewed the diff

[carbolymer]

Nice!

But it does present some issues with race conditions when recreating.

Do you observe isues in nix / hydra? If not, I'd happily approve - otherwise it's a blocker.

We have GHA tests limited to 1 thread, so that shouldn't be an issue there I think.

[palas]

Nice!

But it does present some issues with race conditions when recreating.

Do you observe isues in nix / hydra? If not, I'd happily approve - otherwise it's a blocker.

We have GHA tests limited to 1 thread, so that shouldn't be an issue there I think.

I haven't seen any issues in CI because of what you say that it is limited to 1 thread. It does make the RECREATE_GOLDEN_FILES=1 crash consistently if you enable multithreading (even though it does create the files correctly)

[carbolymer]

But tests in hydra aren't limited to 1 thread, are they? But we're not using RECREATE_GOLDEN_FILES=1 so they should work from what you're saying.

[carbolymer]

I think in the end I'm against the change. Making tests unreliably fail in the local environment will be very confusing for anyone working on the codebase.

[palas]

But tests in hydra aren't limited to 1 thread, are they? But we're not using RECREATE_GOLDEN_FILES=1 so they should work from what you're saying.

Exactly

I think in the end I'm against the change. Making tests unreliably fail in the local environment will be very confusing for anyone working on the codebase.

100%, I could try to find some simple solution to that if that is the only issue

[palas]

@carbolymer, turns out fixing the issue with RECREATE_GOLDEN_FILES=1 was really easy: c19b61d 😎

[carbolymer]

@palas Nice catch! If it makes your solution stable in all cases, then I'm approving.

[Jimbo4350]

Nice 👍

Before optimization: 28.315 s
After Optimization: 20.663 s

Squash the commits

[Jimbo4350]

Nice 👍