Running coverlet.msbuild inside docker with --no-build

[pankti11]

I am trying to run coverlet.msbuild with --no-build flag inside the docker container. And I am never able to generate the code coverage. I cannot run restore in the container. Hence, container does not have an access coverlet.msbuild nuget.

Logs:
Command:
dotnet test $startDir/$testPath/$testFile /p:CollectCoverage=true /p:CoverletOutputFormat=json /p:CoverletOutput=$outputFile /p:MergeWith=$mergeFile --verbosity diag --no-build

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed LogDataConverterMapsLogLevelsToLogSeverities [15 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (None,Unknown) [9 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Trace,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Debug,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Information,Information) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Warning,Warning) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Error,Error) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Critical,Error) [< 1 ms]
  Passed LogDataConverterDetectsUnknownLogLevelMappings [< 1 ms]
  Passed LogStoreProviderEnumDefinedTest [148 ms]
  Passed LogStoreProviderEnumNotDefinedTest [< 1 ms]

Test Run Successful.
Total tests: 10
     Passed: 10
 Total time: 1.1038 Seconds
                   Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask". (TaskId:21)
21:17:54.857     1>Done building target "VSTest" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:2)
21:17:54.858     1>Done Building Project "/github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (VSTest target(s)).
Deferred Messages
21:17:54.861     0>
Detailed Build Summary
======================
     (TaskId:0)
                     
                     ============================== Build Hierarchy (IDs represent configurations) =====================================================
                     Id                  : Exclusive Time   Total Time   Path (Targets)
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     0                   : 1.768s           1.768s       /github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj (VSTest)  (TaskId:0)
                     
                     ============================== Node Utilization (IDs represent configurations) ====================================================
                     Timestamp:            1        Duration   Cumulative
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     637509214730885310:   0        1.772s     1.772s ................................... (TaskId:0)
                     -----------------------------------------------------------------------------------------------------------------------------------
                     Utilization:          100.0    Average Utilization: 100.0 (TaskId:0)

Project Evaluation Performance Summary:
      290 ms  /github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls

Project Performance Summary:
     1428 ms  /github/workspace/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls
               1428 ms  VSTest                                     1 calls

Target Performance Summary:
        9 ms  ShowCallOfVSTestTaskWithParameter          1 calls
     1423 ms  VSTest                                     1 calls

Task Performance Summary:
        8 ms  Message                                   18 calls
       20 ms  CallTarget                                 1 calls
     1386 ms  Microsoft.TestPlatform.Build.Tasks.VSTestTask   1 calls

Build succeeded.
    0 Warning(s)
    0 Error(s)

Expected Behaviour

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Passed LogDataConverterMapsLogLevelsToLogSeverities [22 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (None,Unknown) [12 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Trace,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Debug,Verbose) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Information,Information) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Warning,Warning) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Error,Error) [< 1 ms]
  Passed LogDataConverterMapsLogLevelsToLogSeverities (Critical,Error) [< 1 ms]
  Passed LogDataConverterDetectsUnknownLogLevelMappings [< 1 ms]
  Passed LogStoreProviderEnumDefinedTest [196 ms]
  Passed LogStoreProviderEnumNotDefinedTest [< 1 ms]

Test Run Successful.
Total tests: 10
     Passed: 10
 Total time: 1.1466 Seconds
                   Done executing task "Microsoft.TestPlatform.Build.Tasks.VSTestTask". (TaskId:22)
12:58:07.009     1>Done building target "VSTest" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:5)
12:58:07.009     1>Target "GenerateCoverageResult: (TargetId:7)" in file "/home/admin1/.nuget/packages/coverlet.msbuild/3.0.3/build/coverlet.msbuild.targets" from project "/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (target "GenerateCoverageResultAfterTest" depends on it):
                   Using "Coverlet.MSbuild.Tasks.CoverageResultTask" task from assembly "/home/admin1/.nuget/packages/coverlet.msbuild/3.0.3/build/coverlet.msbuild.tasks.dll".
                   Task "Coverlet.MSbuild.Tasks.CoverageResultTask" (TaskId:23)
                     Task Parameter:ThresholdStat=minimum (TaskId:23)
                     Task Parameter:ThresholdType=line,branch,method (TaskId:23)
                     Task Parameter:InstrumenterState=/tmp/tmp0vqEQf.tmp (TaskId:23)
                     Task Parameter:OutputFormat=json (TaskId:23)
                     Task Parameter:Threshold=0 (TaskId:23)
                     Task Parameter:Output=/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/coverage.json (TaskId:23)

Calculating coverage result...
                     [coverlet] Hit file '/tmp/COM.iX.Logging.Provider_6c1ec4a4-7989-466c-b501-b83278e7731f' deleted (TaskId:23)
  Generating report '/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/coverage.json'

+-----------------------------+------+--------+--------+
| Module                      | Line | Branch | Method |
+-----------------------------+------+--------+--------+
| COM.iX.Logging.Provider | 80%  | 62.5%  | 66.66% |
+-----------------------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 80%  | 62.5%  | 66.66% |
+---------+------+--------+--------+
| Average | 80%  | 62.5%  | 66.66% |
+---------+------+--------+--------+

                     Output Item(s): 
                         CoverletReport=
                             /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/coverage.json
                                     Format=json (TaskId:23)
                   Done executing task "Coverlet.MSbuild.Tasks.CoverageResultTask". (TaskId:23)
12:58:07.261     1>Done building target "GenerateCoverageResult" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:7)
12:58:07.261     1>Target "GenerateCoverageResultAfterTest: (TargetId:8)" in file "/home/admin1/.nuget/packages/coverlet.msbuild/3.0.3/build/coverlet.msbuild.targets" from project "/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (entry point):
12:58:07.261     1>Done building target "GenerateCoverageResultAfterTest" in project "COM.iX.Logging.Provider.Test.csproj".: (TargetId:8)
12:58:07.262     1>Done Building Project "/home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj" (VSTest target(s)).
Deferred Messages
12:58:07.266     0>
Detailed Build Summary
======================
     (TaskId:0)
                     
                     ============================== Build Hierarchy (IDs represent configurations) =====================================================
                     Id                  : Exclusive Time   Total Time   Path (Targets)
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     0                   : 2.658s           2.658s       /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj (VSTest)  (TaskId:0)
                     
                     ============================== Node Utilization (IDs represent configurations) ====================================================
                     Timestamp:            1        Duration   Cumulative
                     ----------------------------------------------------------------------------------------------------------------------------------- (TaskId:0)
                     637508230846046671:   0        2.661s     2.661s ..................................................... (TaskId:0)
                     -----------------------------------------------------------------------------------------------------------------------------------
                     Utilization:          100.0    Average Utilization: 100.0 (TaskId:0)

Project Evaluation Performance Summary:
      350 ms  /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls

Project Performance Summary:
     2256 ms  /home/admin1/actions-runner/ix-logging/_work/iX-logging/iX-logging/COM.iX.Logging.Provider.Test/COM.iX.Logging.Provider.Test.csproj   1 calls
               2256 ms  VSTest                                     1 calls

Target Performance Summary:
        0 ms  InstrumentModulesNoBuild                   1 calls
        0 ms  GenerateCoverageResultAfterTest            1 calls
        3 ms  ShowCallOfVSTestTaskWithParameter          1 calls
       12 ms  GenerateProgramFile                        1 calls
      252 ms  GenerateCoverageResult                     1 calls
      572 ms  InstrumentModules                          1 calls
     1414 ms  VSTest                                     1 calls

Task Performance Summary:
        2 ms  Message                                   18 calls
        8 ms  CallTarget                                 1 calls
      251 ms  Coverlet.MSbuild.Tasks.CoverageResultTask   1 calls
      567 ms  Coverlet.MSbuild.Tasks.InstrumentationTask   1 calls
     1399 ms  Microsoft.TestPlatform.Build.Tasks.VSTestTask   1 calls

Build succeeded.
    0 Warning(s)
    0 Error(s)

[MarcoRossignoli]

hi @pankti11 sorry for the delay, very busy days, outside container does it work?

[pankti11]

Yes. but the only difference is the outside container I do a build. However, inside the container, I don't do-build. However, the bin folder (dlls are present from the build outside the container) is present in the container. So when I do dotnet test with --nobuild I expect it to generate the code coverage. Maybe I am missing something.

[petli]

Do you copy the source code into the docker container in addition to the binaries? Coverlet skips projects with PDBs but no local source files as a heuristic to filter out libraries that shouldn't be part of the coverage. (See https://github.com/coverlet-coverage/coverlet/blob/master/src/coverlet.core/Instrumentation/Instrumenter.cs#L88)

[petli]

It may also be an issue that the path name of the directory containing the code in the docker image is different to the path name outside it, which means that the source files cannot be found even if they are copied into the image. You can try to enable verbose logging (see https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Troubleshooting.md) and see if it logs messages about not finding the source files.

[pankti11]

So inside the container there is source file as well as the binaries. From the logs I could decipher that the Build Task which is responsible to generate the CodeCoverage is never runs. Hence, I concluded that when I do dotnet test --no-build, dotnet is not aware of the coverlet because I did not run dotnet build. When I do dotnet build inside container and then do dotnet test --no-build then the coverage is created by coverlet without any issue.

[TFTomSun]

@petli We are running into similar issues. Actually we want to build and pack the binaries in one job and consume the nuget packages in multiple test jobs. Would be optimal if we wouldn't have to map the sources in the test job. Is that a requirement?

Additionally we tried as a workaround to map the sources along with the test run, but even that doesn't work. The path to the sources is identically, but of course we didn't do a build in the test job. Is there some output expected relative to the sources which we need to transfer from our build job to the test jobs? (maybe some files in the obj folder? )

Can I somehow configure to instrument binaries with embedded pdbs, even if no sources for them are available?
We have the sources embdeded, along with the embedded pdb. Are embedded sources not supported?

btw. we are using coverlet.collector

[github-actions]

This issue is stale because it has been open for 3 months with no activity.

[github-actions]

This issue was closed because it has been inactive for 9 months since being marked as stale.