False positives from tests/test_self.py #10450
Description
I noticed that some tests in test_self.py fail to validate the behavior they are meant to verify due to the way with pytest.raises(SystemExit): is being used blocking the asserts from ever being run.
For example in tests/test_self.py::TestCallbackOptions::test_errors_only you can change
def test_errors_only() -> None:
"""Test the --errors-only flag."""
with pytest.raises(SystemExit):
run = Run(["--errors-only"])
assert run.linter._error_mode
to
def test_errors_only() -> None:
"""Test the --errors-only flag."""
with pytest.raises(SystemExit):
run = Run(["--an-invalid-option"])
assert run.linter._error_mode
and it still passes perfectly happily but when you look at the output there were errors
$ pytest tests/test_self.py::TestCallbackOptions::test_errors_only -s
============================================================= test session starts =============================================================
platform win32 -- Python 3.12.3, pytest-8.3.5, pluggy-1.5.0
rootdir: D:\Repos\pylint
configfile: pyproject.toml
plugins: anyio-4.9.0, langsmith-0.4.1
collected 1 item
tests\test_self.py usage: pylint [options]
pylint: error: Unrecognized option found: an-invalid-option
.
============================================================== 1 passed in 0.12s ==============================================================
You can even just change the assert line to assert False and it will still pass every time.
This affects multiple tests in test_self.py I confirmed this behavior in test_errorrs_only and test_verbose.