[CI] MinioRepositoryAnalysisRestIT testRepositoryAnalysis failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #9547 / oraclelinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #9498 / rocky-8_platform-support-unix

Reproduction Line:

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:minio:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.MinioRepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=2D5E933E73DCF75C -Dtests.locale=ki-Latn-KE -Dtests.timezone=America/Belem -Druntime.java=24

Applicable branches:
9.0

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]

Issue Reasons:

• [9.0] 2 failures in test testRepositoryAnalysis (1.2% fail rate in 165 executions)
• [9.0] 2 failures in pipeline elasticsearch-periodic-platform-support (25.0% fail rate in 8 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

Hmm this looks very much like a repeat of #122670, there are MPUs that remain in existence after they've been aborted:

[2025-07-23T03:36:13,600][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:13,601][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:13,601][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:13,601][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:13,603][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:13,604][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:13,611][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:13,612][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:28,626][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:28,626][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:28,626][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:28,627][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:28,628][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:28,629][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:28,629][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmQxNzBkZGViLTU2OTQtNGZiYy04NDc4LWQ2YWViMDRjNjE0YngxNzUzMjUyNTczNjE0MDAwNDI4] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:28,630][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmQxNzBkZGViLTU2OTQtNGZiYy04NDc4LWQ2YWViMDRjNjE0YngxNzUzMjUyNTczNjE0MDAwNDI4] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:28,630][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:28,631][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmQxNzBkZGViLTU2OTQtNGZiYy04NDc4LWQ2YWViMDRjNjE0YngxNzUzMjUyNTczNjE0MDAwNDI4] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:28,631][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQwY2YwOTJkLTM0M2YtNDUyMi1iY2M5LWMwNjI5M2RkOGY5OXgxNzUzMjUyNTczNjE0NDU2NjQ5] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:28,632][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQwY2YwOTJkLTM0M2YtNDUyMi1iY2M5LWMwNjI5M2RkOGY5OXgxNzUzMjUyNTczNjE0NDU2NjQ5] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:28,633][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQwY2YwOTJkLTM0M2YtNDUyMi1iY2M5LWMwNjI5M2RkOGY5OXgxNzUzMjUyNTczNjE0NDU2NjQ5] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:28,637][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]   
[2025-07-23T03:36:28,639][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmQxNzBkZGViLTU2OTQtNGZiYy04NDc4LWQ2YWViMDRjNjE0YngxNzUzMjUyNTczNjE0MDAwNDI4] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:28,643][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQwY2YwOTJkLTM0M2YtNDUyMi1iY2M5LWMwNjI5M2RkOGY5OXgxNzUzMjUyNTczNjE0NDU2NjQ5] initiated at [2025-07-23T03:36:13.614-0300]   
[2025-07-23T03:36:43,665][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:43,668][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:43,668][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LmYyNmU5YzZiLTJmNDItNGU5Mi04ZTkzLWZiNWFmOTk0NWZhZngxNzUzMjUyNTU4NTg5NzQyNzE2] initiated at [2025-07-23T03:35:58.589-0300]   
[2025-07-23T03:36:43,671][WARN ][o.e.r.s.S3BlobContainer  ] [test-cluster-0] cleaning up stale compare-and-swap upload [MmVmMThlMWEtMmU0YS00OTQ0LWIxZjktN2UxMDNhYTZmMDQ3LjQxNzZmYWQ2LWVhMWUtNDNhMS05ZjEyLWZiN2QxZjBkYjZlMngxNzUzMjUyNTU4NTkyNTYyNjA4] initiated at [2025-07-23T03:35:58.592-0300]  

I'll see if I can reproduce this more simply and open another MinIO issue.

[DaveCTurner]

Relatedly #131744 extends the HTTP timeout so we'll get something a little more useful than just java.net.SocketTimeoutException: 60,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE] in future.

[DaveCTurner]

This is another MinIO bug, very similar to the previous one. I opened minio/minio#21456. I think we can reinstate the workaround from #127166 to keep these tests passing.