Builtin Container Registry: Manifests 404 Not Found

[fekle]

Description

Hi there!

I'm in the process of migrating from GitLab to Gitea, and so far I'm very happy with Gitea - migration was smooth, and things like Actions work great!

Now I wanted to also use Gitea as my Docker Registry to host my container images, but I'm stuck there. Pushing image layers works fine, but the final manifest push fails with 404. I think this issue is related to existing issues such as #31802 and #31644, but I didn't want to spam those issues.

First of all - to make sure that this is not related to my migrated groups and repositories - I created a new Organization called "test" and a project called "test" in my Gitea instance. I also enabled debug logging.

Then, when I'm trying to push a minimal image from my computer, I get this:

> cat Dockerfile
FROM debian:12
CMD bash

> docker build -t GITEA_HOST/test/test:test .
[+] Building 1.1s (5/5) FINISHED
 => [internal] load build definition from Dockerfile
[...]
 => => naming to GITEA_HOST/test/test:test
 => => unpacking to GITEA_HOST/test/test:test

> docker push GITEA_HOST/test/test:test
The push refers to repository [GITEA_HOST/test/test]
7a351543ef85: Pushed
7b24851aa36d: Pushed
failed commit on ref "index-sha256:[...]": unexpected status from PUT request to https://GITEA_HOST/v2/test/test/manifests/test: 404 Not Found

The same thing happens with an Actions workflow and BuildKit.

On the Gitea server side, here are the startup logs:

Server listening on :: port 22.
Server listening on 0.0.0.0 port 22.
2024/08/18 11:33:09 cmd/web.go:242:runWeb() [I] Starting Gitea on PID: 19
2024/08/18 11:33:09 cmd/web.go:111:showWebStartupMessage() [I] Gitea version: 1.22.1 built with GNU Make 4.4.1, go1.22.5 : bindata, timetzdata, sqlite, sqlite_unlock_notify
[...]
2024/08/18 11:33:09 cmd/web.go:304:listen() [I] Listen: http://0.0.0.0:3000
2024/08/18 11:33:09 cmd/web.go:308:listen() [I] AppURL(ROOT_URL): https://GITEA_HOST/
2024/08/18 11:33:09 cmd/web.go:311:listen() [I] LFS server enabled
2024/08/18 11:33:09 ...s/graceful/server.go:50:NewServer() [I] Starting new Web server: tcp:0.0.0.0:3000 on PID: 19
2024/08/18 11:33:09 ...s/graceful/server.go:63:func1() [D] Starting server on tcp:0.0.0.0:3000 (PID: 19)

And this is the output when pushing aforementioned image:

2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 401 Unauthorized in 0.1ms @ container/container.go:126(container.ReqContainerAccess)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 401 Unauthorized in 0.1ms @ container/container.go:126(container.ReqContainerAccess)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 401 Unauthorized in 0.1ms @ container/container.go:126(container.ReqContainerAccess)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 401 Unauthorized in 0.0ms @ container/container.go:126(container.ReqContainerAccess)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/token for REDACTED:0, 404 Not Found in 0.0ms @ container/container.go:172(container.AuthenticateNotImplemented)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed GET /v2/token?scope=%2A&scope=repository%3Atest%2Ftest%3Apull%2Cpush&service=container_registry for REDACTED:0, 200 OK in 35.2ms @ container/container.go:149(container.Authenticate)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 404 Not Found in 1.6ms @ container/container.go:486(container.HeadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 404 Not Found in 1.7ms @ container/container.go:486(container.HeadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 404 Not Found in 2.1ms @ container/container.go:486(container.HeadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDACTED:0, 404 Not Found in 2.1ms @ container/container.go:486(container.HeadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/test/test/blobs/uploads/ for REDACTED:0, 202 Accepted in 0.6ms @ container/container.go:218(container.InitiateUploadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/test/test/blobs/uploads/ for REDACTED:0, 202 Accepted in 0.7ms @ container/container.go:218(container.InitiateUploadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/test/test/blobs/uploads/ for REDACTED:0, 202 Accepted in 0.9ms @ container/container.go:218(container.InitiateUploadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/test/test/blobs/uploads/ for REDACTED:0, 202 Accepted in 0.9ms @ container/container.go:218(container.InitiateUploadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/blobs/uploads/jtzjr5ghfg6qfr6heaqmg0mlh?digest=sha256%3A[...] for REDACTED:0, 201 Created in 2.1ms @ container/container.go:373(container.EndUploadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/blobs/uploads/49vudymsvznarmnzzeaumpg7g?digest=sha256%3A[...] for REDACTED:0, 201 Created in 3.4ms @ container/container.go:373(container.EndUploadBlob)
2024/08/18 11:41:32 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/blobs/uploads/ada4izfsmp6vozywuvtpwuq3t?digest=sha256%3A[...] for REDACTED:0, 201 Created in 4.1ms @ container/container.go:373(container.EndUploadBlob)
2024/08/18 11:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/blobs/uploads/dnvgyyksiuqixcv3ajfxygxfz?digest=sha256%3A[...] for REDACTED:0, 201 Created in 496.1ms @ container/container.go:373(container.EndUploadBlob)
2024/08/18 11:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/manifests/test for REDACTED:0, 404 Not Found in 0.9ms @ container/container.go:624(container.HeadManifest)
2024/08/18 11:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/manifests/test for REDACTED:0, 404 Not Found in 0.9ms @ container/container.go:624(container.HeadManifest)
2024/08/18 11:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/manifests/test for REDACTED:0, 201 Created in 3.3ms @ container/container.go:539(container.UploadManifest)
2024/08/18 11:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/manifests/test for REDACTED:0, 201 Created in 4.9ms @ container/container.go:539(container.UploadManifest)
2024/08/18 11:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/manifests/test for REDACTED:0, 200 OK in 0.5ms @ container/container.go:624(container.HeadManifest)
2024/08/18 11:41:33 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/manifests/test for REDACTED:0, 404 Not Found in 0.6ms @ container/container.go:539(container.UploadManifest)

Strangely, I'm seeing 401 Unauthorized errors there; which I'm not seeing in the Docker client.
I am logged into the registry via docker login with my administrator user. I also tried creating a new user, which I added to the aforementioned test group, but the same issue persisted.
I also made sure that things like filesystem permissions as well as the reverse proxy configuration are correct.

For some context, here Is my app.ini

APP_NAME = REDACTED
RUN_MODE = prod
RUN_USER = git
WORK_PATH = /data/gitea

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
DOMAIN = GITEA_HOST
SSH_DOMAIN = GITEA_HOST
HTTP_PORT = 3000
ROOT_URL = https://GITEA_HOST/
DISABLE_SSH = false
SSH_PORT = 222
SSH_LISTEN_PORT = 22
LFS_START_SERVER = true
LFS_JWT_SECRET = REDACTED
OFFLINE_MODE = true

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD =
LOG_SQL = false
SCHEMA =
SSL_MODE = disable
SQLITE_JOURNAL_MODE = WAL

[indexer]
REPO_INDEXER_ENABLED = true
REPO_INDEXER_PATH = /data/gitea/indexers/repos.bleve
ISSUE_INDEXER_ENABLED = true
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars

[attachment]
PATH = /data/gitea/attachments

[log]
MODE = console
LEVEL = debug
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK = true
SECRET_KEY =
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = REDACTED
PASSWORD_HASH_ALGO = argon2

[service]
DISABLE_REGISTRATION = true
REQUIRE_SIGNIN_VIEW = true
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost

[lfs]
PATH = /data/git/lfs

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = false
ENABLE_OPENID_SIGNUP = false

[cron.update_checker]
ENABLED = false

[repository.pull-request]
DEFAULT_MERGE_STYLE = merge

[repository.signing]
DEFAULT_TRUST_MODEL = committer

[oauth2]
JWT_SECRET = REDACTED

[actions]
ENABLED = true

[packages]
ENABLED = true

The reverse proxy configuration:

GITEA_HOST {
	reverse_proxy http://gitea:3000
}

And the docker compose config for Gitea:

name: gitea

services:
  gitea:
    container_name: gitea
    image: gitea/gitea:1
    pull_policy: always
    restart: always
    init: true
    hostname: "GITEA_HOST"
    networks:
      - gitea
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: "4G"
    environment:
      USER_UID: "3000"
      USER_GID: "3000"
    ports:
      - "222:22"
    volumes:
      - "REDACTED:/data:rw"

networks:
  gitea:
    name: gitea
    external: true

I'm kind of at a loss here and would be very grateful if anyone could point me in the right direction :-)

Gitea Version

1.22.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.45.2

Operating System

Docker image: gitea/gitea:1; Host: Debian 12.6 6.1.0-23-cloud-amd64

How are you running Gitea?

Hosting myself, on Docker 27.0.3 behind Caddy v2.8.4.

Database

SQLite

[ChristopherHX]

Please try docker build -t GITEA_HOST/test/test:test . --provenance=false, then push.

Based on my research this could have to do with build provenance aka https://docs.docker.com/build/attestations/slsa-provenance/.

My docker setup is old and has no containerd image store, therefore is my system probably unaffected as I can push. For me this seems to be turned off by default due to that reason and trying to enable it tells me that my setup cannot attest.

docker build -t GITEA_HOST/test/test:test . --attest type=provenance,mode=max
ERROR: Attestation is not supported for the docker driver.
Switch to a different driver, or turn on the containerd image store, and try again.
Learn more at https://docs.docker.com/go/attestations/

I would try to debug this next week with provenance enabled for a dind container or similar.

maybe I'm on a wrong path here.

[fekle]

Thanks for your swift reply!

I just tried building and pushing without provenance, that seems to work, with plain docker at least - thanks!
Pushing with Buildkit via earthly unfortunately still fails, see below.

> docker build --provenance=false -t GITEA_HOST/test/test:test-without-provenance .                                                               
[+] Building 0.5s (5/5) FINISHED                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                         
[...]                                                                                 
 => => unpacking to GITEA_HOST/test/test:test-without-provenance                                                                                       

> docker push GITEA_HOST/test/test:test-without-provenance                                                                                       
The push refers to repository [GITEA_HOST/test/test]
7b24851aa36d: Layer already exists
test-without-provenance: digest: sha256:[...] size: 503

(fun fact: at first the size: 503 confused me, thinking that now i'm getting an error 503 ...)

Gitea log:

2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDCACTED:0, 401 Unauthorized in 0.1ms @ container/container.go:126(container.ReqContainerAccess)
2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDCACTED:0, 401 Unauthorized in 0.0ms @ container/container.go:126(container.ReqContainerAccess)
2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed POST /v2/token for REDCACTED:0, 404 Not Found in 0.0ms @ container/container.go:172(container.AuthenticateNotImplemented)
2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed GET /v2/token?scope=%2A&scope=repository%3Atest%2Ftest%3Apull%2Cpush&service=container_registry for REDCACTED:0, 200 OK in 57.1ms @ container/container.go:149(container.Authenticate)
2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDCACTED:0, 200 OK in 1.3ms @ container/container.go:486(container.HeadBlob)
2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/blobs/sha256:[...] for REDCACTED:0, 200 OK in 1.2ms @ container/container.go:486(container.HeadBlob)
2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed HEAD /v2/test/test/manifests/test-without-provenance for REDCACTED:0, 404 Not Found in 0.5ms @ container/container.go:624(container.HeadManifest)
2024/08/18 14:35:13 ...eb/routing/logger.go:102:func1() [I] router: completed PUT /v2/test/test/manifests/test-without-provenance for REDCACTED:0, 201 Created in 3.6ms @ container/container.go:539(container.UploadManifest)

Note that the first HEAD requests still results in 401, do you know what's up with that?
That's also what is now breaking my BuildKit/Earthly build:

unexpected status from HEAD request to https://GITEA_HOST/v2/test/test/blobs/sha256:[...]: 401 Unauthorized

Also, there is still a 404 error for that manifest endpoint, but the docker client is not complaining anymore - strange.

Please let me know if i can be of any help with debugging this issue further :)

[fekle]

I had a look at where the 401 is coming from, container/container.go:126; thats:

	if ctx.Doer == nil || (setting.Service.RequireSignInView && ctx.Doer.IsGhost()) {
		apiUnauthorizedError(ctx)
	}

As a quick test, i set REQUIRE_SIGNIN_VIEW to false in my app.ini, restarted Gitea and retried, but i'm still getting the same error with Buildkit - so that's not it. Given that this issue seems to come from ctx.Doer == nil and occurs at the HEAD request, maybe Buildkit is not sending any credentials with that particular request?

[ChristopherHX]

what is now breaking my BuildKit/Earthly build

Are those still have provenance on? I mean it's not unlikely that provenance is not supported yet by the Gitea Container Registry.

See
https://docs.docker.com/build/building/variables/#buildx_no_default_attestations for an env variable.

Also, there is still a 404 error for that manifest endpoint, but the docker client is not complaining anymore - strange.

Pure logical, isn't the 404 here just telling docker that the tag doesn't exist yet
Then docker creates the tag with PUT that returns created. Now I would expect doing this a second time wouldn't result in a 404 before the put

BTW yes I used buildkit to push my image yesterday, no error (what I tested is in discord, between your post and your reply to me)

My Log

::group::GitHub Actions runtime token ACs
: write
::endgroup::
::group::Docker info
[command]/usr/bin/docker version
Client:
 Version:           26.1.4-1
 API version:       1.45
 Go version:        go1.21.11
 Git commit:        5650f9b10226d75e8e9a490a31cc3e5b846e0034
 Built:             Wed Jun  5 10:47:13 UTC 2024
 OS/Arch:           linux/arm64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:39:57 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[command]/usr/bin/docker info
Client:
 Version:    26.1.4-1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.15.1-1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.28.1-1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 42
  Running: 9
  Paused: 0
  Stopped: 33
 Images: 311
 Server Version: 26.1.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-1055-raspi
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.624GiB
 Name: ubuntu
 ID: 2bbda9e8-6738-44c4-8348-144dfa28c073
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 97
  Goroutines: 111
  System Time: 2024-08-17T21:56:03.805054279Z
  EventsListeners: 0
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
::endgroup::
::group::Proxy configuration
No proxy configuration found
::endgroup::
::group::Buildx version
[command]/usr/bin/docker buildx version
github.com/docker/buildx 0.15.1-1 1c1dbb2e4cb5363110f42102744a08d034c2300d
::endgroup::
::group::Builder info
{
  "nodes": [
    {
      "name": "default",
      "endpoint": "default",
      "status": "running",
      "buildkit": "v0.13.2",
      "platforms": "linux/arm64,linux/arm/v7,linux/arm/v6",
      "features": {
        "Automatically load images to the Docker Engine image store": true,
        "Cache export": true,
        "Docker exporter": true,
        "Multi-platform build": true,
        "OCI exporter": true
      },
      "labels": {
        "org.mobyproject.buildkit.worker.moby.host-gateway-ip": "172.17.0.1"
      }
    }
  ],
  "name": "default",
  "driver": "docker"
}
::endgroup::
[command]/usr/bin/docker buildx build --iidfile /tmp/docker-actions-toolkit-IqcWDl/build-iidfile-d9dbbc13a2.txt --tag domain/christopher/test-image:develop --metadata-file /tmp/docker-actions-toolkit-IqcWDl/build-metadata-6189bffe81.json --push .
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 55B done
#1 DONE 0.0s
#2 [internal] load metadata for docker.io/library/ubuntu:latest
#2 DONE 0.0s
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [1/1] FROM docker.io/library/ubuntu:latest
#4 CACHED
#5 exporting to image
#5 exporting layers done
#5 writing image sha256:d8398bc503e6b06179ed7c2416aa3a837c3d9ba2093f4d6a5521d991f33767be done
#5 naming to domain/christopher/test-image:develop done
#5 DONE 0.0s
#6 resolving provenance for metadata file
#6 DONE 0.0s
#7 pushing domain/christopher/test-image:develop with docker
#7 pushing layer a510ae0f066c
#7 pushing layer a510ae0f066c 4.33MB / 69.25MB 1.1s
#7 pushing layer a510ae0f066c 8.61MB / 69.25MB 1.8s
#7 pushing layer a510ae0f066c 12.49MB / 69.25MB 2.5s
#7 pushing layer a510ae0f066c 16.87MB / 69.25MB 3.2s
#7 pushing layer a510ae0f066c 20.64MB / 69.25MB 3.8s
#7 pushing layer a510ae0f066c 24.43MB / 69.25MB 4.3s
#7 pushing layer a510ae0f066c 28.81MB / 69.25MB 5.1s
#7 pushing layer a510ae0f066c 32.69MB / 69.25MB 5.8s
#7 pushing layer a510ae0f066c 36.51MB / 69.25MB 6.6s
#7 pushing layer a510ae0f066c 40.37MB / 69.25MB 7.3s
#7 pushing layer a510ae0f066c 44.72MB / 69.25MB 8.0s
#7 pushing layer a510ae0f066c 48.56MB / 69.25MB 8.7s
#7 pushing layer a510ae0f066c 52.41MB / 69.25MB 9.4s
#7 pushing layer a510ae0f066c 56.67MB / 69.25MB 10.2s
#7 pushing layer a510ae0f066c 60.38MB / 69.25MB 11.6s
#7 pushing layer a510ae0f066c 64.10MB / 69.25MB 12.2s
#7 pushing layer a510ae0f066c 67.84MB / 69.25MB 13.1s
#7 pushing layer a510ae0f066c 71.59MB / 69.25MB 13.8s
#7 pushing layer a510ae0f066c 16.3s done
#7 DONE 16.7s
::group::ImageID
sha256:d8398bc503e6b06179ed7c2416aa3a837c3d9ba2093f4d6a5521d991f33767be
::endgroup::
::group::Digest
sha256:c77a24d4a20eef7c039be023f4c82bfd4c00a2bf82dd318785cd97312835aa67
::endgroup::
::group::Metadata
{
  "buildx.build.provenance": {
    "buildType": "https://mobyproject.org/buildkit@v1",
    "materials": [
      {
        "uri": "pkg:docker/ubuntu@latest?platform=linux%2Farm64",
        "digest": {
          "sha256": "a510ae0f066c65d73f523d5821fe048a6858e53360ecb8d39b7b931162f11479"
        }
      }
    ],
    "invocation": {
      "configSource": {
        "entryPoint": "Dockerfile"
      },
      "parameters": {
        "frontend": "dockerfile.v0",
        "locals": [
          {
            "name": "context"
          },
          {
            "name": "dockerfile"
          }
        ]
      },
      "environment": {
        "platform": "linux/arm64"
      }
    }
  },
  "buildx.build.ref": "default/default/ayj5ct2nam1a1pvtqze1rmy32",
  "containerimage.config.digest": "sha256:d8398bc503e6b06179ed7c2416aa3a837c3d9ba2093f4d6a5521d991f33767be",
  "containerimage.digest": "sha256:c77a24d4a20eef7c039be023f4c82bfd4c00a2bf82dd318785cd97312835aa67",
  "image.name": "domain/christopher/test-image:develop"
}
::endgroup::
::group::Reference
default/default/ayj5ct2nam1a1pvtqze1rmy32
::endgroup::
::group::Check build summary support
Build summary is not yet supported on GHES
::endgroup::

[ChristopherHX]

Ok I'm out, as I cannot reproduce the issue on my system, provenance push didn't fail for me.

I have created an dind container with containerd image cache and it doesn't fail.

The flag REQUIRE_SIGNIN_VIEW isn't enabled for me, because my ports are not exposed to the www and my reverse proxy configuration is probably totally different.

Maybe someone else can have a look

[fekle]

Are those still have provenance on? I mean it's not unlikely that provenance is not supported yet by the Gitea Container Registry.

Yes, i used BUILDX_NO_DEFAULT_ATTESTATIONS=1 but could well be that buildkit itself is not picking up that argument, as i'm using a buildkit in my CI; I will have to debug this further. Kinda annoying that BuildKit does not handle this gracefully and just assumes that the registry supports this feature.
But the 401 in my log output above is from a normal docker push with the --provenance=false command, which seems to ignore the 401 when BuildKit doesn't. Given that there provenance is disabled for sure, that seems to be a different issue.

Pure logical, isn't the 404 here just telling docker that the tag doesn't exist yet
Then docker creates the tag with PUT that returns created. Now I would expect doing this a second time wouldn't result in a 404 before the put

While that would indeed make sense, I don't think that's the case as that still occurs when the image and all layers already exist, and the error in the logs is container.AuthenticateNotImplemented. Not sure what that is about, but it could be related to the 401s before.

[fekle]

Ok I'm out, as I cannot reproduce the issue on my system, provenance push didn't fail for me.

I have created an dind container with containerd image cache and it doesn't fail.

The flag REQUIRE_SIGNIN_VIEW isn't enabled for me, because my ports are not exposed to the www and my reverse proxy configuration is probably totally different.

Maybe someone else can have a look

That's interesting that it doesnt fail for you, so maybe it's something with my configuration. But thank you for your help!

[fekle]

Allright, I got pushing in Actions to work.

I was using

- name: login to container registry
  uses: docker/login-action@v2
  with:
    registry: GITEA_HOST
    username: ${{ gitea.actor }}
    password: ${{ gitea.token }}

to log in to the container registry.

While the login succeeds, pushes fail. When I create a personal access token with access to packages and then use my username as username as well as that token as password, I can push.

Now, if I want to keep that more secure, I can put that token into a secret - however, given that I can only define secrets per repository and not globally, I'll have to copy that secret definition to every repository that I'm using CI with.

Is there a better way to do this? Researching on Google and so on leads to scattered forum posts from various versions, so I can't seem to find a definitive answer. Coming from GitLab CI, there were 3 auto-generated variables in CI: CI_REGISTRY, CI_REGISTRY_USER, and CI_REGISTRY_PASSWORD. That made pushing to the built-in registry a breeze; it would be nice if Gitea had something similar - I would think that pushing to the built-in registry is a common action in CI.

I'd be happy to write some documentation on that and open a PR once I know the "idiomatic" way to do all of this :)

[shaikzoheb-oi]

@fekle In my case, the organisation is "oi" and project is "test". It's a similar use-case like you, where I am trying to use gitea as container registry along with git repo. In my scenario, I've deployed gitea locally and somehow it's failing.

% docker build -t 127.0.0.1:3000/oi/test .
[+] Building 2.1s (5/5) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 61B 0.0s
=> [internal] load metadata for docker.io/library/debian:12 2.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [1/1] FROM docker.io/library/debian:12@sha256:b8084b1a576c5504a031936e1132574f4ce1d6cc7130bbcc25a28f074539ae6b 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:0c21f32bd1037ae97c3535bb9c7f15f7ca0f829e5809161f607531c20764b61e 0.0s
=> => naming to 127.0.0.1:3000/oi/test 0.0s

% docker push 127.0.0.1:3000/oi/test:test
The push refers to repository [127.0.0.1:3000/oi/test]
Get "http://127.0.0.1:3000/v2/": dial tcp 127.0.0.1:3000: connect: connection refused

telnet and netcat is working:

nc -zv 127.0.0.1 3000
Connection to 127.0.0.1 port 3000 [tcp/hbci] succeeded!

telnet 127.0.0.1 3000
Trying 127.0.0.1...
Connected to localhost.

[jcjveraa]

Exact same issue here. Running the latest official Gitea image via docker. The main git functionality has been working fine for months. The comments above seem to suggest creating a PAT can work, but that didn't do it for me. I created a new PAT, and tried a docker push. The PAT was used according to the Gitea UI, but still the same output as others see.

Then tried a docker build --provenance=false image build, which pushed fine.

I had expected this to work somewhat more 'out of the box' as a basic push doesn't seem special.

For reference, output of the failed push below. Again: with --provenance=false worked fine. Replaced strings with descriptions for privacy.

$ docker --version
Docker version 27.3.1, build ce12230

$ docker push [host]/[organization]/[repo]/[image-name]:latest
The push refers to repository [[host]/[organization]/[repo]/[image-name]]
73b263454379: Pushed 
d88d75d6bbae: Layer already exists 
a480a496ba95: Layer already exists 
a5072a025aa2: Layer already exists 
ae8522101371: Layer already exists 
88900f0244bb: Layer already exists 
5cfa0738ac2f: Layer already exists 
2f779fda9095: Layer already exists 
24f2c78e0691: Layer already exists 
failed commit on ref "index-sha256:07761bba483df9b7123227be97a913b01bb62bee3533214537d694ec3902ec08": unexpected status from PUT request to https://[host]/v2/[organization]/[repo]/[image-name]/manifests/latest: 404 Not Found

[lunny]

What's your PAT scope?

[jcjveraa]

Good point - (non public) packages read & write, I failed to consider a wider scope but on the flip side thinking about it now I would expect that to be enough. Op do 24 okt 2024 om 00:18 schreef Lunny Xiao ***@***.***>

…

What's your PAT scope? — Reply to this email directly, view it on GitHub <#31861 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6CPHKPBZT6ZYDUTJ4F5L3Z5AOBZAVCNFSM6AAAAABMWJQFWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZTGU3TAOJXHE> . You are receiving this because you commented.Message ID: ***@***.***>

[sbrown61]

I'm having a similar issue with podman push. As far as I know, there is no equivalent for --provenance=false. Podman version is 5.2.5 and gitea is 1.22.3 (via docker hub):

Getting image source signatures
Copying blob 49cadc25f7bb [--------------------------------------] 8.0b / 1.4MiB | 1.9 MiB/s
Copying blob edeb1fd6bf66 [--------------------------------------] 8.0b / 4.1MiB | 1.7 MiB/s
Copying blob 8cbe4b54fa88 [--------------------------------------] 8.0b / 80.1MiB | 1.1 MiB/s
Copying blob fa3e1fa14e04 [--------------------------------------] 8.0b / 341.5KiB | 2.4 MiB/s
Copying blob 7ddf8331309f [--------------------------------------] 8.0b / 3.5KiB | 4.0 MiB/s
Copying blob dcd40c14ad70 [--------------------------------------] 8.0b / 9.1MiB | 1.1 MiB/s
Copying blob 9b03e049cc12 [--------------------------------------] 8.0b / 157.3MiB | 1.2 MiB/s
Error: writing blob: initiating layer upload to /v2/[USERNAME]/blobs/uploads/ in [HOST]:[PORT]: StatusCode: 404, ""