- These are arbitrary length limits that had a load of disables in code.
- The limits were only increasing over time rather than decreasing.
- Fixing the problematic code to be shorter would take a long time for
questionable gain since the problem has been around so long.
- We're not going to make the really long things be any shorter any time soon.
- The instructions in issue 14685 say, pragmatically, "disable all the rubocop
rules we're never going to realistically fix e.g. Metrics/ClassLength". But
that felt like a slippery slope to more _really_ long modules/classes/blocks,
and the limits are here for a reason.
- Extract update tests into separate tests.
- Make `brew tests --online` run only once in its own job. This job
could be made non-required to fight flakiness.
- Split up the various macOS runs into several steps now that we have
more parallel macOS workers available.
- Cleanup some flaky tests.
- fix `check_for_config_scripts` test, no idea why this wasn't working
- don't autoretry tests if `focus` or `byebug` are enabled
- add a ARM64 Ventura bottle symlink
BuildPulse is trying to find flaky tests for us but, given the previous
model of using `rspec-retry`, it would rarely find them. Instead, let's
try to always rerun `brew tests` multiple times, report to BuildPulse
each time (by moving the reporting logic into `brew tests`) and disable
`rspec-retry` when using BuildPulse.
While we're here, let's enable `rspec-retry` locally so we don't have
flaky tests biting maintainers/contributors there.
The `VERBOSE_TESTS` variable was from cask and never gets set (and
is unset by `bin/brew`). Replace it with `HOMEBREW_VERBOSE_TESTS` and
set it by `--verbose` or `--debug`.
While we're here, remove an unneeded `VERBOSE` delete (as it's already
done by `bin/brew`).
- always retry each test at least once (confusingly this means a retry
count of 2 rather than 1)
- always wait at least 1 second between retries
- set a default retry metadata for integration tests rather than
overriding any specified values
- use `example.run` rather than `example.run_with_retry` for integration
tests because, confusingly, this avoids having the retry count be
half what it should be (because the attempts increases by one for
each `run_with_retry` call)
- use 4 retries for integration tests with 2**attempts*retry_wait
(retry wait now being 2) to actually have more retries than before
this commit (but with exponential backoff)
This should generally improve test flakiness in CI but particularly
improve the cleanup test flake we've seen recently.
