Build server + local docker registry stuck on retrying after download?

[GabZanMacaw]

Hello! I'm running a Coolify instance with a Docker registry and a separate build server that uses that registry. So far, it'd been working without issue. However there is one specific image that refuses to work. The image is built out of a Dockerfile, it finishes building successfully, is pushed to the registry, downloaded, no errors in the registry logs, but then when it gets to the "Download completed" or "Verifying checksum" part of it, the deploy starts retrying a bunch and then gives up with no more information. What is it even retrying? How can I get more information out of this so I can see what's going on?

A few things I checked before posting:

1. I have tried redeploying other apps with the same build server + private registry configuration and they worked.
2. I checked the storage on both localhost (50GB free) and the build server (14GB free)
3. I tried manually deleting the image on the registry, doing a system prune, and pushing again, same problem.
4. The domain is behind a Cloudflare proxy (not my domain, don't have access to it) and right now it's giving me a Cloudflare 502 Bad Gateway error on the host, the last change I made was changing the domain and redeploying, I don't know if that's relevant? There is a previous commit of the website that is accessible, I can see it on the sslip.io link, but Cloudflare seems to be having an issue with it. EDIT: The domain simply wasn't pointing to the Coolify server, client screwed up, but that didn't fix the issue unfortunately.

[djsisson]

@GabZanMacaw are both your servers behind cloudflare proxy?

If an image layer is above 100mb then cf will block it with a 413

Solutions are

1. alter dockerfile
2. don't use cf proxy
3. use private networking and not go through cf to pull image
4. pay cf to increase limit

[GabZanMacaw]

No, only that specific website was using Cloudflare proxy on its domain. My localhost and build server don't use any proxy. I found out that the client had screwed up on the DNS registers so that's why I was getting a 503, it wasn't pointing to the right server, but that still didn't fix the retrying problem unfortunately. I had to disable the build server and then that website built just fine.

[djsisson]

you can test, running docker pull registry/image:tag on the website server

[GabZanMacaw]

Sorry for the wait, I tried pulling on the website server and it printed:

Error response from daemon: manifest for registry.macawbrasil.com.br/macaw/casa-madeira-website:latest not found: manifest unknown: manifest unknown

I checked the catalog and the image is there and it has two tags, but I don't know how to check what's wrong with the manifests.