[SYCL] Detect ze call leaking in E2E tests

[jiezzhang]

After #19328, UR_L0_LEAKS_DEBUG stop throwing exceptions when leaks are detected so LIT can't report failures. Add a leak checking in format.py to keep "--param ur_l0_leaks_debug=1" work as before.

[cperkinsintel]

Just FYI, we already have a lit var %{l0_leak_check} which a bunch of the tests still use (instead of UR_L0_LEAKS_DEBUG ). It gets replaced by the env var in the final invocation, but I'm not sure if this python script here will detect.

I think that l0_leak_check directive is no longer needed and probably could be replaced by the actual env var. It's in hundreds of tests right now.

[igchor]

I think the tests that set the env var (either directly or by using %{l0_leak_check}) already check for absence of "LEAK" (--implicit-check-not=LEAK) so we don't have to check it again here. As far as I understand this is only needed when the user decides to run all the tests with leak checking (e.g. passing --param ur_l0_leaks_debug=1 to lit).

[jiezzhang]

Pretty sure the failed test is not caused by this PR. @cperkinsintel is it a known flaky issue?
SYCL :: Reduction/reduction_internal_nd_range_1dim.cpp

[igchor]

Pretty sure the failed test is not caused by this PR. @cperkinsintel is it a known flaky issue? SYCL :: Reduction/reduction_internal_nd_range_1dim.cpp

I don't know if it;s a known issue but it seems unrelated to the PR

[igchor]

@intel/llvm-reviewers-runtime could you please take a look the PR?

[aelovikov-intel]

After #19328, UR_L0_LEAKS_DEBUG stop throwing exceptions

Why does "unification" cause that?

[igchor]

After #19328, UR_L0_LEAKS_DEBUG stop throwing exceptions

Why does "unification" cause that?

The leak checking is now happening in the loader, during loader teardown, and so there is no place for us to throw the exception from anymore.

[aelovikov-intel]

The leak checking is now happening in the loader, during loader teardown, and so there is no place for us to throw the exception from anymore.

Because it's C and not C++? Can it abort?

[igchor]

The leak checking is now happening in the loader, during loader teardown, and so there is no place for us to throw the exception from anymore.

Because it's C and not C++? Can it abort?

Mostly because it's done in the library destructor which means we don't have an entry point from which we could return an error (we had urAdapterTeardown when leak checking was done in UR).

We could abort() but leaks are not really a critical failure so I think just parsing the output in tests is a better option.

[aelovikov-intel]

The leak checking is now happening in the loader, during loader teardown, and so there is no place for us to throw the exception from anymore.

Because it's C and not C++? Can it abort?

Mostly because it's done in the library destructor which means we don't have an entry point from which we could return an error (we had urAdapterTeardown when leak checking was done in UR).

We could abort() but leaks are not really a critical failure so I think just parsing the output in tests is a better option.

Can we have one more env variable control to request that abort? I think that would still be much better than parsing output (that might be redirected and not available for parsing).

[igchor]

The leak checking is now happening in the loader, during loader teardown, and so there is no place for us to throw the exception from anymore.

Because it's C and not C++? Can it abort?

Mostly because it's done in the library destructor which means we don't have an entry point from which we could return an error (we had urAdapterTeardown when leak checking was done in UR).
We could abort() but leaks are not really a critical failure so I think just parsing the output in tests is a better option.

Can we have one more env variable control to request that abort? I think that would still be much better than parsing output (that might be redirected and not available for parsing).

@nrspruit What do you think about calling abort? Do you think there is any other way to report the leaks?