Make `--nocapture` optional

[kpreid]

When running tests, rust-analyzer unconditionally adds --nocapture to the test arguments. I would like this to be optional, because using --nocapture leads to sometimes-unreadable interleaved output, especially when two tests fail simultaneously, or when successful tests are producing verbose output intended to be read only if the test fails.

[dnut]

I would implement this by making cargo args configurable in general, allowing the user to enter an arbitrary string. There's nothing particularly special about --nocapture that entitles it to special treatment compared to other cargo args. I would also make it empty by default, as in --nocapture should not be enabled unless the user explicitly turns it on by typing it, due to the reasoning in #13040.

[kpreid]

A possible complication in "just" making test arguments customizable: there are both arguments to cargo and arguments to the test binary, and users might want to specify one or both. I think accepting a custom argument list that may include "--" should solve that problem adequately and intuitively.

[SludgePhD]

This feature would be great to have. The current behavior is opaque and the presence of --nocapture is never communicated to the user. Neither is the unconditionally set RUST_BACKTRACE=short.

This makes test suites that make use of #[should_panic] difficult to use with rust-analyzer because all the panic messages and backtraces make the output completely unreadable:

    Blocking waiting for file lock on build directory
   Compiling zaru-linalg v0.1.0 (/home/sludge/code/Zaru/crates/zaru-linalg)
    Finished test [unoptimized] target(s) in 0.43s
     Running unittests src/lib.rs (target/debug/deps/zaru_linalg-57770df66f31a28a)

running 3 tests
thread 'matrix::view::tests::view_oob' panicked at crates/zaru-linalg/src/matrix/view.rs:96:9:
assertion failed: c_off + VC <= self.num_columns()
stack backtrace:
test matrix::view::tests::compare ... ok
test matrix::view::tests::view ... ok
   0: rust_begin_unwind
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:127:5
   3: zaru_linalg::matrix::view::MatrixView<T,R,C>::fixed_view
   4: zaru_linalg::matrix::view::tests::view_oob
   5: zaru_linalg::matrix::view::tests::view_oob::{{closure}}
   6: core::ops::function::FnOnce::call_once
   7: core::ops::function::FnOnce::call_once
             at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
test matrix::view::tests::view_oob - should panic ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 42 filtered out; finished in 0.01s

(this output is with only a single #[should_panic] test)

[Veykril]

Right, RUST_BACKTRACE=short is set for r-a to show better panic info by default, leaking that into cargo invocations is something we should prevent though.

[MingweiSamuel]

--show-output does not show the output if a test is hanging. To re-enable --nocapture use

"rust-analyzer.runnables.extraTestBinaryArgs": [
        "--nocapture"
],

[matklad]

I personally feel strongly that --nocapture is the right default. The current default of --show-output makes it impossible to debug the tests, as you don't see the output until the end, which might not come.

The more "stuff" happens between the user and the test binary, the harder it is to realize what's happening.

The problem with interleaved test output should be fixed by making sure that passing tests do not print anything.

Interleaving output is a feature when you debug a tricky concurrency bug caused by interraction between the tests.

It is of course file to have an option to use --show-output as an easy-accessible checkbox, but I am 0.8 sure that --nocapture would be the right default leading to better ecosystem, precisely because it forces you to be intentional about your test's output.

To reiterate, this is my personal opinion, and, while I do hold is strongly, it could be wrong.