Spurious timeout in dist builders #48827
Description
This is a reopening of #48192 now that the sccache issue identified in that issue has been fixed. The dist builders are still occasionally timing out over three hours, and we need to get that under control!
In a discussion today some various possibilities arose, but unfortunately there doesn't seem to be many silver bullets here:
- A long-term change could be to perhaps leverage build stages on Travis. Build stages would then allow us to avoid timeouts almost entirely and perhaps even shorten build times having certain sequences of the build be much more parallel than before, leveraging available capacity on Travis. This is of course a pretty significant investment, though, and may be difficult to implement.
- Skip documenting cross-compiled builds altogether. The ARM documentation is probably really similar to the x86 documentation, and skipping docs entirely would skip rustdoc for the host, rustbook, and building the docs themselves. More information is needed though about how much time this would save.
- Precompiling possible tools like rustfmt and Cargo. This doesn't work for tools that depend on rustc/unstable features, but it's possible that we could cross-compile binaries in their own CI systems. The downside of this though is that it spreads out already complicated CI configuration, and the RLS itself depends on both rustfmt/cargo in library form so we may have to build them anyway...
I'll try to keep this updated with known causes of timeouts and possible mitigation strategies.