Support BuildKit

[chris-crone]

Currently the Python SDK performs builds using the older build infrastructure. It would be good to support BuildKit so that users of the SDK can benefit from its various advantages.

[chris-crone]

The current known blocker for this is we need a Python implementation of the fssync gRPC service (a service which runs on the session but on the client side) and that service is distinctly non-trivial to implement.

[chris-crone]

Request for build secret support: #2174

[haizaar]

Request for build secret support: #2230

Did you mean to link to my original ticket? :)

[chris-crone]

Fixed, thanks @haizaar

[AkihiroSuda]

The current known blocker for this is we need a Python implementation of the fssync gRPC service (a service which runs on the session but on the client side) and that service is distinctly non-trivial to implement.

In addition to fssync, there are auth, content, secrets, and sshforward client-side services.

As @mipearson commented in moby/buildkit#685 , docker-py should just invoke docker build CLI.

[tonistiigi]

My recommendation is to create a helper binary that compose can call as a quick practical solution. Note that fssync is optional for BuildKit, the Docker API can still receive tarballs fine and build them with BuildKit. It's less optimal but you still get the other benefits. But if you want secrets/ssh support then you do need either a helper or reimplement the grpc provider.

[AkihiroSuda]

Do we need a helper binary other than existing docker CLI?

[tonistiigi]

There are some things that might be interesting for compose that can be done in Go library and not in CLI atm. Like doing multiple requests in parallel with a shared session.

[mipearson]

Is there a possible future where most docker-compose things are done via a helper binary rather than via docker-py?

I seem to be having a very bad run of hitting docker-py specific issues lately :(