OCI Registry uses `AppURL` (ROOT_URL) no matter which address the client is connecting to (more Multi-Domain fallout)

[jessesanford]

Description

No matter what url the OCI client uses, at first connection Gitea responds with the appropriate www-authenticate header but with an incorrect token url hostname set to the AppURL instead of the hostname from the client's request.

gitea/routers/api/packages/container/container.go

Line 118 in c660149

ctx.Resp.Header().Add("WWW-Authenticate", `Bearer realm="`+setting.AppURL+`v2/token",service="container_registry",scope="*"`)

This means that we cannot use OCI clients that target protocol://host:port combinations other than the one set in ROOT_URL as they are redirected by the www-auth header to the protocol://host:port in the ROOT_URL via it's use in the construction of the AppURL variable.

The use case is familiar to anyone who is using gitea to back an on-kubernetes-cluster registry to host images that are pushed from off-cluster but referenced for resources on-cluster.

Very often the ingress protocol://host:port combination that is fronting Gitea as a reverse proxy is inaccessible to on cluster resources or containerized resources (like the kubelet in a kind setup). So those resources will be configured to connect directly to the service bound to the Gitea deployment and the protocol://host:port combination that the kubelet connects to on cluster will not match the one used off cluster.

While much work was done in #19345 to fix the issues with multi-domain support in the gui, it did not make any effort to solve the same problem in the container registry. Therefore #22033 should probably not have been closed.

FWIW the docker registry project uses the request headers to set the authorization response header here I believe https://github.com/distribution/distribution/blob/51a72c2aef976bd55de3a7b8b0120f97b4169476/internal/client/auth/challenge/authchallenge.go#L119-L121 so there is precedent for this behavior.

Gitea Version

1.21.7 built with GNU Make 4.4.1, go1.21.7 : bindata, timetzdata, sqlite, sqlite_unlock_notify

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

KO pushing to Gitea from developer workstation

# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/04 10:58:00 Using base cgr.dev/chainguard/static:latest@sha256:67ed8ca8d99e12e8778c038cf88ef7c27d44f08247d317c7135a66ca9d8a7652 for ucp.adskeng.net/unified-control-plane
2024/03/04 10:58:00 Building ucp.adskeng.net/unified-control-plane for linux/amd64
2024/03/04 10:58:05 Publishing gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5:latest
2024/03/04 10:58:05 pushed blob: sha256:dddad3d1675134ab45c0e954718630bd68921d716701511f0afb7ca2f37b61e0
2024/03/04 10:58:06 pushed blob: sha256:c28efc4d3128372f019baf58f3d4ccce238a4a5ebe583983f329875815f3796d
2024/03/04 10:58:06 pushed blob: sha256:250c06f7c38e52dc77e5c7586c3e40280dc7ff9bb9007c396e06d96736cf8542
2024/03/04 10:58:06 pushed blob: sha256:f254e377e71e3753dc3b88a86088ea46dc949f0ae3b50434e7a1f3b2608c8b73
2024/03/04 10:58:06 pushed blob: sha256:0d64d3736fe6666f37553b9deb11783bdbef4e63301b939dc3c51a8aa2eb5a6d
2024/03/04 10:58:06 gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5:sha256-9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e.sbom: digest: sha256:7bbe466320ab724e1e7a0e83bb8be736fae0769ca9d9399a000af66ab5d5bee9 size: 375
2024/03/04 10:58:06 Published SBOM gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5:sha256-9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e.sbom
2024/03/04 10:58:06 pushed blob: sha256:9a6dad177a5c7758bc5ebba82a4dfeca298cd068b29c0cfb3641264e42bf500e
2024/03/04 10:58:06 gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5:latest: digest: sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e size: 1211
2024/03/04 10:58:06 Published gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e

Pod referencing Gitea Image on Kind cluster that is also hosting Gitea

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  34s                default-scheduler  Successfully assigned unified-control-plane-system/unified-control-plane-controller-manager-59645d95b8-w2bg8 to localdev-control-plane
  Normal   Pulling    33s                kubelet            Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1"
  Normal   Pulled     31s                kubelet            Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1" in 1.920296393s (1.920351294s including waiting)
  Normal   Created    31s                kubelet            Created container kube-rbac-proxy
  Normal   Started    31s                kubelet            Started container kube-rbac-proxy
  Normal   Pulling    17s (x2 over 31s)  kubelet            Pulling image "gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e"
  Warning  Failed     17s (x2 over 31s)  kubelet            Failed to pull image "gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e": rpc error: code = Unknown desc = failed to pull and unpack image "gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e": failed to resolve reference "gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e": failed to authorize: failed to fetch anonymous token: Get "https://gitea.cnoe.localtest.me:8443/v2/token?scope=%2A&scope=repository%3Agiteaadmin%2Funified-control-plane-856ee3c8576196fe1db39ad1b44799b5%3Apull&service=container_registry": dial tcp 127.0.0.1:8443: connect: connection refused
  Warning  Failed     17s (x2 over 31s)  kubelet            Error: ErrImagePull
  Normal   BackOff    2s (x3 over 30s)   kubelet            Back-off pulling image "gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e"
  Warning  Failed     2s (x3 over 30s)   kubelet            Error: ImagePullBackOff

Git Version

No response

Operating System

No response

How are you running Gitea?

Deployed with Helm on Kind

% helm install --dry-run my-gitea gitea-charts/gitea --values pkg/controllers/localbuild/resources/gitea/values.yaml > pkg/controllers/localbuild/resources/gitea/k8s/install.yaml

% cat pkg/controllers/localbuild/resources/gitea/values.yaml
redis-cluster:
  enabled: false
postgresql:
  enabled: false
postgresql-ha:
  enabled: false

persistence:
  enabled: false

test:
  enabled: false

gitea:
  admin:
    existingSecret: gitea-admin-secret
  config:
    database:
      DB_TYPE: sqlite3
    session:
      PROVIDER: memory
    cache:
      ADAPTER: memory
    queue:
      TYPE: level
    server:
      DOMAIN: gitea.cnoe.localtest.me
      ROOT_URL: 'https://gitea.cnoe.localtest.me:{{ .Port }}'

service:
  ssh:
    type: NodePort
    nodePort: 32222
    externalTrafficPolicy: Local

ingress:
  enabled: true
  apiVersion: 'networking.k8s.io/v1'
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
  className: nginx
  hosts:
    - host: gitea.cnoe.localtest.me
      paths:
        - path: /
          pathType: Prefix

Relevant containerd config.toml from the Kind node with rewrite to make kubelet pull from the container port rather than the host port of the ingress:

version = 2

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."gitea.cnoe.localtest.me"]
          [plugins."io.containerd.grpc.v1.cri".registry.configs."gitea.cnoe.localtest.me".tls]
            insecure_skip_verify = true
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gitea.cnoe.localtest.me:8443"]
          endpoint = ["https://gitea.cnoe.localtest.me"]

Solution Proof of Concept

As a proof of concept I created a branch over here to test if I could influence the current behavior:

https://github.com/jessesanford/gitea/blob/c2fffabcd5122d258360d4443b6e583fc62e28f1/routers/api/packages/container/container.go#L119-L131

ko pushing container image to gitea.cnoe.localtest.me:8443 from local workstation to on-cluster gitea:

% export KO_DOCKER_REPO=gitea.cnoe.localtest.me:8443/giteaadmin/ && kustomize build config/overlays/local | ko resolve --insecure-registry -f - | kubectl apply -f -
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/04 18:02:42 Using base cgr.dev/chainguard/static:latest@sha256:67ed8ca8d99e12e8778c038cf88ef7c27d44f08247d317c7135a66ca9d8a7652 for ucp.adskeng.net/unified-control-plane
2024/03/04 18:03:05 Building ucp.adskeng.net/unified-control-plane for linux/amd64
2024/03/04 18:03:09 Publishing gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5:latest
2024/03/04 18:03:10 existing manifest: latest@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e
2024/03/04 18:03:10 existing manifest: sha256-9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e.sbom@sha256:7bbe466320ab724e1e7a0e83bb8be736fae0769ca9d9399a000af66ab5d5bee9
2024/03/04 18:03:10 Published SBOM gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5:sha256-9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e.sbom
2024/03/04 18:03:10 Published gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e
namespace/unified-control-plane-system unchanged
customresourcedefinition.apiextensions.k8s.io/certificates.tls.ucp.adskeng.net unchanged
customresourcedefinition.apiextensions.k8s.io/gitops.registry.ucp.adskeng.net unchanged
customresourcedefinition.apiextensions.k8s.io/sloes.observability.ucp.adskeng.net configured
serviceaccount/unified-control-plane-controller-manager unchanged
role.rbac.authorization.k8s.io/unified-control-plane-leader-election-role unchanged
clusterrole.rbac.authorization.k8s.io/unified-control-plane-manager-role configured
clusterrole.rbac.authorization.k8s.io/unified-control-plane-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/unified-control-plane-proxy-role unchanged
rolebinding.rbac.authorization.k8s.io/unified-control-plane-leader-election-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/unified-control-plane-manager-rolebinding unchanged
clusterrolebinding.rbac.authorization.k8s.io/unified-control-plane-proxy-rolebinding unchanged
service/unified-control-plane-controller-manager-metrics-service unchanged
deployment.apps/unified-control-plane-controller-manager configured

Pod events from deploy of pod referencing the container image at gitea.cnoe.localtest.me:8443 pushed by ko:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  76s   default-scheduler  Successfully assigned unified-control-plane-system/unified-control-plane-controller-manager-59645d95b8-65zk7 to localdev-control-plane
  Normal  Pulling    75s   kubelet            Pulling image "gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1"
  Normal  Pulled     73s   kubelet            Successfully pulled image "gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1" in 1.891202544s (1.891260145s including waiting)
  Normal  Created    73s   kubelet            Created container kube-rbac-proxy
  Normal  Started    73s   kubelet            Started container kube-rbac-proxy
  Normal  Pulling    73s   kubelet            Pulling image "gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e"
  Normal  Pulled     72s   kubelet            Successfully pulled image "gitea.cnoe.localtest.me:8443/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5@sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e" in 1.408522344s (1.408531044s including waiting)
  Normal  Created    71s   kubelet            Created container manager
  Normal  Started    71s   kubelet            Started container manager

NOTE: the kubelet thinks that it is getting the container from gitea.cnoe.localtest.me:8443 but containerd is pulling it from gitea.cnoe.localtest.me:443

From gitea logs during pull (See the X-Forwarded-Host:[gitea.cnoe.localtest.me] X-Forwarded-Port:[443] in the debug log message)

2024/03/04 22:46:00 ...ntainer/container.go:118:apiUnauthorizedError() [E] Request looks like: &{GET /v2/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5/manifests/sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e?ns=gitea.cnoe.localtest.me%3A8443 HTTP/1.1 1 1 map[Accept:[application/vnd.oci.image.manifest.v1+json, */*] Accept-Encoding:[gzip] User-Agent:[containerd/v1.7.1] X-Forwarded-For:[127.0.0.1] X-Forwarded-Host:[gitea.cnoe.localtest.me] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Scheme:[https] X-Real-Ip:[127.0.0.1] X-Request-Id:[becf1c5a6a65a6ca507217c9e135639a] X-Scheme:[https]] {} <nil> 0 [] false gitea.cnoe.localtest.me map[] map[] <nil> map[] 10.244.0.7:54542 /v2/giteaadmin/unified-control-plane-856ee3c8576196fe1db39ad1b44799b5/manifests/sha256:9c6a4c54450cc8dd93c445d473da318a037b9535d98f76567d144458b1572a4e?ns=gitea.cnoe.localtest.me%3A8443 <nil> <nil> <nil> 0xc0062ab140 <nil> [] map[]}

Database

None

[KN4CK3R]

Added the packages label because of the specific topic but this issue is not really package related. Gitea simply does not have multi-domain support at the moment.

[jessesanford]

The solution for this case is pretty simple @KN4CK3R. See my POC: https://github.com/jessesanford/gitea/blob/c2fffabcd5122d258360d4443b6e583fc62e28f1/routers/api/packages/container/container.go#L119-L131 Would it be worth crafting a PR for this? What are the chances of this getting approved and merged without a broader multi-domain solution. Given the incremental solutions so far: #19345 it would seem like we should be able to get this going right?

[jessesanford]

@KN4CK3R Sorry to ping you directly, is there another maintainer I should be working with on this? Thank you for your help!

[lunny]

Maybe AppURL is the fallback URL. We don't know the address the docker/podman client requested because maybe there is a proxy between Gitea and client. This is different from web browser, we can know the web browser's request domain.

[jessesanford]

Maybe AppURL is the fallback URL.

FYI for those following this: https://github.com/jessesanford/gitea/pull/1/files#diff-36426f1aaff58f9b2084f7fc13593b237474e1d75a4e2883e0a44351e367ff76R125

Do you think that we would check for X-Forwarded-* headers and if they are not available then fall back on setting.AppURL rather than ctx.Req.URL.Host ? I am fine with that. I think in most instances they will be the same thing, as the proxy SHOULD have its origin for gitea is configured to the same setting.AppURL address. As long as we can use the X-Forwarded-* headers when present I am happy. They are pretty standard for most reverse proxy and api firewalls at this point.

[KN4CK3R]

@wxiaoguang You did some work removing root url occurences. I don't know what other multi url systems do and what are best practices. For example I don't know if it is a good idea to allow every url the client passes in or if we need an allow list or something.

[wxiaoguang]

The "X-Forwarded-* header" approach works for many cases, I didn't use it because the FAQ there: #22861

Why not use {{.GuessCurrentOrigin $.ctx ...}} to let backend decide the absolute URL?

It's difficult for backend to guess the correct protocol(scheme) correctly with zero configuration. Generating the absolute URL from frontend can guarantee that the URL is 100% correct -- since the user is visiting it.

According to MDN, the X-Forwarded-Proto is not standard and there are many alternatives (Front-End-Https / X-Forwarded-Protocol / X-Forwarded-Ssl / X-Url-Scheme). Gitea should be able to be easy to setup and use out-of-box.

---

So, for advanced users who could configure their system correctly, I think it is feasible and useful to use X-Forwarded-* headers, the only thing need to do is to introduce some config options and help end users to use them correctly.

Update: to keep simple, maybe hard code X-Forwarded-Proto(Scheme) / X-Forwarded-Host for the first step.

[jessesanford]

Ok I will craft a PR based on this feedback and then we can continue the discussion there.