- Never bother to do `brew cleanup` after `brew install`
- Skip `brew test-bot --only-cleanup-before` in more places it's not
needed
- Move `brew config`/`brew doctor` into the default formula run
(through `brew test-bot --only-setup`)
- Install `buildpulse-test-reporter` rather than letting `brew tests`
do it
- Improve the formatting of the `brew tests` step
- Further shorten the code coverage filenames
- Don't output BuildPulse results upload status unless failed
- Extract update tests into separate tests.
- Make `brew tests --online` run only once in its own job. This job
could be made non-required to fight flakiness.
- Split up the various macOS runs into several steps now that we have
more parallel macOS workers available.
- Cleanup some flaky tests.
- remove all places we check the flag so it's a no-op now
- flip the messaging to refer to `HOMEBREW_NO_INSTALL_FROM_API` where
relevant
- adjust the documentation
- Change name of rubocop warning
- Disable linting on remaining offending lines
- Add todos to move lines with disabled linting
checks to extend/os in the future
BuildPulse is trying to find flaky tests for us but, given the previous
model of using `rspec-retry`, it would rarely find them. Instead, let's
try to always rerun `brew tests` multiple times, report to BuildPulse
each time (by moving the reporting logic into `brew tests`) and disable
`rspec-retry` when using BuildPulse.
While we're here, let's enable `rspec-retry` locally so we don't have
flaky tests biting maintainers/contributors there.
In preparation for detecting flaky tests with BuildPulse, this commit
sets up the rspec_junit_formatter gem to output JUnit XML reports of the
test suite, which is the format used by BuildPulse and various other
tooling that interprets test results.
Because the test suite uses the parallel_tests gem, this commit
incorporates some related changes to make all the parallel_tests gem and
the rspec_junit_formatter gem to cooperate with each other.
rspec_junit_formatter writes everything to a single XML file. That works
fine when there's only one process writing to the file. By default,
whatever process finishes last will write to the file and clobber the
output of all the other processes that wrote to the file. 🙈
To prevent this issue, the parallel_tests wiki recommends adding a
`.rspec_parallel` file to specify its RSpec options
(https://github.com/grosser/parallel_tests/wiki#with-rspec_junit_formatter----by-jgarber),
then the project can specify different files for each process to write
to like so:
--format RspecJunitFormatter
--out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
However, prior to this commit, the Homebrew/brew test suite specified
its RSpec options via the command line. Unfortunately though, there's no
way (AFAICT) to set the equivalent of these options via the command
line:
--format RspecJunitFormatter
--out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml
So, we need to use a `.rspec_parallel` file to specify these options ☝️.
However, it appears that RSpec allows you to specify formatters _either_
in an options file (like `.rspec_parallel`) _or_ via command-line args.
But if you specify any formatters via command-line args, then all
formatters in the options file are ignored. (I suspect that's somehow
related to this bit of code in rspec-core:
https://github.com/rspec/rspec-core/blob/v3.10.0/lib/rspec/core/configuration_options.rb#L64.)
With that in mind, in order to have the RspecJunitFormatter configured
in `.rspec_parallel`, we need to move the other formatters into
`.rpsec_parallel` as well, instead of passing them as command-line
args. Therefore, this commit moves all the formatters into a
`.rspec_parallel` file.
The `VERBOSE_TESTS` variable was from cask and never gets set (and
is unset by `bin/brew`). Replace it with `HOMEBREW_VERBOSE_TESTS` and
set it by `--verbose` or `--debug`.
While we're here, remove an unneeded `VERBOSE` delete (as it's already
done by `bin/brew`).
