Container registry - layer drops

[PavelMXFox]

Description

Hi!
The layer 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 is constantly disappearing, it contains 1024 zero-bytes. The 'gitea/packages/4f/4f' folder remains empty. If I upload an archived version of a layer, everything works until any image containing this layer is deleted, and after that the layer disappears. At this time in the gitea logs

2022/05/02 21:50:41 [62702801] router: completed GET /v2/mxfox/fox-web-basic/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for 217.107.125.158:0, 500 Internal Server Error in 255.8ms @ container/container.go:359(container.GetBlob)

2022/05/02 21:50:42 ...ntainer/container.go:84:api Error() [E] [62702802] open /data/gitea/packages/4f/4f/4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: no such file or directory

I attach a copy of the layer.

Gitea Version

1.17.0+dev-511-g71bafa026

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

in docker

Database

MySQL

[PavelMXFox]

Update - this repeated with other blobs, when pushing other image, contains this blob. Blob's file was removed from FS, but in DB table package_blob row with it presents, and because it blob cant'be re-uploaded with docker push

[PavelMXFox]

Can't reproduce on [1.17.0+dev-568-gcbd45471b] it seems to be fixed.

[PavelMXFox]

Gitea [1.17.0+dev-587-g9ea920640]
Image layer has been dropeed by registry.

Directory packages/2f/f2 is empty.

if i restore 2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd file from backup - all work fine.

It was when i build && push image with docker buildx
Docker's log

#13 exporting to image
#13 exporting manifest list sha256:42c0b4a9357d56f15b69abfc9e8cc7b3488a62a1f9cb6444d3817afdd75efcb9 0.0s done
#13 pushing layers
#13 3.635 error: failed commit on ref "layer-sha256:fabe278fb707a6ec1d19f0a2eb628261cde71a98b6f00c89cb437a561a5097ac": unexpected status: 500 Internal Server Error
#13 3.635 retrying in 1s
#13 3.668 error: failed commit on ref "layer-sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd": unexpected status: 500 Internal Server Error

Gitea log

2022/05/18 17:21:27 ...ntainer/container.go:84:apiError() [E] [628500e7-7] open /data/gitea/packages/2f/f2/2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd: no such file or directory
2022/05/18 17:21:27 [628500e7-7] router: completed GET /v2/mxfox/fox-web-basic/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for __:0, 500 Internal Server Error in 40.0ms @ container/container.go:358(container.GetBlob)
2022/05/18 17:22:58 ...ntainer/container.go:84:apiError() [E] [62850142-4] open /data/gitea/packages/2f/f2/2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd: no such file or directory
2022/05/18 17:22:58 [62850142-4] router: completed GET /v2/mxfox/fox-web-basic/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for __:0, 500 Internal Server Error in 64.1ms @ container/container.go:358(container.GetBlob)
2022/05/18 17:26:41 ...ntainer/container.go:84:apiError() [E] [62850221-7] open /data/gitea/packages/2f/f2/2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd: no such file or directory
2022/05/18 17:26:41 [62850221-7] router: completed GET /v2/mxfox/fox-web-basic/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for __:0, 500 Internal Server Error in 76.0ms @ container/container.go:358(container.GetBlob)
2022/05/18 17:26:41 ...ntainer/container.go:84:apiError() [E] [62850221-9] open /data/gitea/packages/2f/f2/2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd: no such file or directory
2022/05/18 17:26:41 [62850221-9] router: completed GET /v2/mxfox/fox-web-basic/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for __:0, 500 Internal Server Error in 32.3ms @ container/container.go:358(container.GetBlob)
2022/05/18 17:26:46 ...ntainer/container.go:84:apiError() [E] [62850226] open /data/gitea/packages/2f/f2/2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd: no such file or directory
2022/05/18 17:26:46 [62850226] router: completed GET /v2/mxfox/fox-web-basic/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for __:0, 500 Internal Server Error in 34.1ms @ container/container.go:358(container.GetBlob)
2022/05/18 17:26:46 ...ntainer/container.go:84:apiError() [E] [62850226-2] open /data/gitea/packages/2f/f2/2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd: no such file or directory
2022/05/18 17:26:46 [62850226-2] router: completed GET /v2/mxfox/fox-web-basic/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for __:0, 500 Internal Server Error in 33.6ms @ container/container.go:358(container.GetBlob)

[PavelMXFox]

I see series of HTTP500, usually after some layers drop. Now I can't find if something layers are missing, maybe it's a new bug. Just in case, I attach a full push log.

#13 3.925 error: failed commit on ref "layer-sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac": unexpected status: 500 Internal Server Error

2022/05/18 20:49:55 [628531c3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034 for ____:0, 401 Unauthorized in 0.3ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 401 Unauthorized in 0.4ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:659bbfc26a5a93b8010ea6b575f7f0a60cdfe6a80e5e154fde755479d301f0ce for ____:0, 401 Unauthorized in 1.2ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 401 Unauthorized in 0.3ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-5] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:39e7b07e7410ac8be4d4d7e7de8a2cf0a214278d15410d6eac5828595ab02085 for ____:0, 401 Unauthorized in 0.4ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-6] router: completed POST /v2/token for ____:0, 405 Method Not Allowed in 0.2ms @ packages/api.go:254(packages.ContainerRoutes)
2022/05/18 20:49:55 [628531c3-7] router: completed GET /v2/token?scope=repository%3Amxfox%2Fchimera-mk2-core%3Apull%2Cpush for ____:0, 200 OK in 263.2ms @ container/container.go:137(container.Authenticate)
2022/05/18 20:49:56 [628531c4-4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:39e7b07e7410ac8be4d4d7e7de8a2cf0a214278d15410d6eac5828595ab02085 for ____:0, 200 OK in 51.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 404 Not Found in 56.6ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 404 Not Found in 62.9ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034 for ____:0, 200 OK in 75.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-6] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 404 Not Found in 48.2ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-5] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:659bbfc26a5a93b8010ea6b575f7f0a60cdfe6a80e5e154fde755479d301f0ce for ____:0, 200 OK in 58.6ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-8] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 140.3ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:56 [628531c4-7] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 249.2ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:56 [628531c4-9] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:02909e51564bd118b45605cf034515fbad616eeb4547d32e8e43c30ce73a507d for ____:0, 200 OK in 258.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-10] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 305.9ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:56 [628531c4-11] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for ____:0, 200 OK in 434.9ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-16] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for ____:0, 200 OK in 23.1ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-14] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:82a003d50dff92ab1725d548b14447af15583ee2f475dfb83cb7582b0026f416 for ____:0, 404 Not Found in 401.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:57 [628531c4-17] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:70dfc71c1e12e4d8a617313a286212d2b3cd0751c3df8c57e9408048d060d956 for ____:0, 200 OK in 360.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:57 [628531c5] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 285.3ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:57 [628531c5-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d5fd17ec1767521cf97f61568096bfc9a7cd9c2d149576a7b43930c5a97062b0 for ____:0, 200 OK in 23.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:57 [628531c4-13] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/k9h6area7tbavls4fewkuetvy?digest=sha256%3A06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 201 Created in 1081.7ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:57 ...es/container/blob.go:56:func1() [E] [628531c4-12] Error inserting package: Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:57 ...es/container/blob.go:56:func1() [E] [628531c4-15] Error inserting package: Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:57 [628531c4-15] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/ro1rkcdp1ib4awzbummsl24oc?digest=sha256%3A09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 500 Internal Server Error in 1077.3ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:57 [628531c4-12] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/dl0e8ctrfrwh1jiu7i6a403y1?digest=sha256%3Afe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 500 Internal Server Error in 1256.4ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:57 ...ntainer/container.go:84:apiError() [E] [628531c4-15] Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:57 ...ntainer/container.go:84:apiError() [E] [628531c4-12] Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:58 [628531c6] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 404 Not Found in 57.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:58 [628531c6-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 404 Not Found in 88.8ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:59 [628531c6-3] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 66.6ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:59 [628531c6-4] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 51.5ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:59 [628531c7] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/95o8db7u0w9snfrt7itn8h5vh?digest=sha256%3A09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 201 Created in 332.7ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:59 [628531c7-2] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/acdmgf4ph173eyp5jcqrgllqk?digest=sha256%3Afe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 201 Created in 390.7ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:59 [628531c5-3] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/ab3ohnddvqlltk8ddwffmxgd4?digest=sha256%3A82a003d50dff92ab1725d548b14447af15583ee2f475dfb83cb7582b0026f416 for ____:0, 201 Created in 2063.3ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:50:00 [628531c8] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:6c8411bdc95928eb58cceca5a543cd0c9137383f48b0d3fa03b41afee57548a6 for ____:0, 404 Not Found in 21.5ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:00 [628531c8-2] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/sha256:6c8411bdc95928eb58cceca5a543cd0c9137383f48b0d3fa03b41afee57548a6 for ____:0, 201 Created in 380.9ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:00 [628531c8-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:d9ef2c79f420f4b83129d006441d00286e3f1a696266b4ef6e97c2017c898662 for ____:0, 404 Not Found in 20.9ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:01 [628531c8-4] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/sha256:d9ef2c79f420f4b83129d006441d00286e3f1a696266b4ef6e97c2017c898662 for ____:0, 201 Created in 364.4ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:01 [628531c9] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/testing-2-c69e9a8f0a for ____:0, 404 Not Found in 19.7ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:01 [628531c9-2] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/testing-2-c69e9a8f0a for ____:0, 201 Created in 111.9ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:01 [628531c9-4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:82a003d50dff92ab1725d548b14447af15583ee2f475dfb83cb7582b0026f416 for ____:0, 200 OK in 77.3ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-5] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034 for ____:0, 200 OK in 86.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:02909e51564bd118b45605cf034515fbad616eeb4547d32e8e43c30ce73a507d for ____:0, 200 OK in 100.8ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-6] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 200 OK in 101.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-7] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:70dfc71c1e12e4d8a617313a286212d2b3cd0751c3df8c57e9408048d060d956 for ____:0, 200 OK in 96.9ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-8] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 200 OK in 77.2ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-10] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 200 OK in 66.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-9] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:659bbfc26a5a93b8010ea6b575f7f0a60cdfe6a80e5e154fde755479d301f0ce for ____:0, 200 OK in 81.1ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-12] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for ____:0, 200 OK in 40.2ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-11] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for ____:0, 200 OK in 91.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-13] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d5fd17ec1767521cf97f61568096bfc9a7cd9c2d149576a7b43930c5a97062b0 for ____:0, 200 OK in 34.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-14] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:39e7b07e7410ac8be4d4d7e7de8a2cf0a214278d15410d6eac5828595ab02085 for ____:0, 200 OK in 36.8ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-15] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:d9ef2c79f420f4b83129d006441d00286e3f1a696266b4ef6e97c2017c898662 for ____:0, 200 OK in 28.4ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:01 [628531c9-16] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:6c8411bdc95928eb58cceca5a543cd0c9137383f48b0d3fa03b41afee57548a6 for ____:0, 200 OK in 22.7ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:02 [628531ca] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/testing for ____:0, 200 OK in 21.8ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:02 [628531ca-2] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/testing for ____:0, 201 Created in 295.6ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:06 [628531ce] router: completed POST /api/v1/repos/Chimera.Mark2/core/statuses/c69e9a8f0a5e35a40d292466236c12740c9db5e2 for ____:0, 201 Created in 345.8ms @ repo/status.go:21(repo.NewCommitStatus)

[lunny]

I see series of HTTP500, usually after some layers drop. Now I can't find if something layers are missing, maybe it's a new bug. Just in case, I attach a full push log.

#13 3.925 error: failed commit on ref "layer-sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac": unexpected status: 500 Internal Server Error

2022/05/18 20:49:55 [628531c3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034 for ____:0, 401 Unauthorized in 0.3ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 401 Unauthorized in 0.4ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:659bbfc26a5a93b8010ea6b575f7f0a60cdfe6a80e5e154fde755479d301f0ce for ____:0, 401 Unauthorized in 1.2ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 401 Unauthorized in 0.3ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-5] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:39e7b07e7410ac8be4d4d7e7de8a2cf0a214278d15410d6eac5828595ab02085 for ____:0, 401 Unauthorized in 0.4ms @ container/container.go:113(container.ReqContainerAccess)
2022/05/18 20:49:55 [628531c3-6] router: completed POST /v2/token for ____:0, 405 Method Not Allowed in 0.2ms @ packages/api.go:254(packages.ContainerRoutes)
2022/05/18 20:49:55 [628531c3-7] router: completed GET /v2/token?scope=repository%3Amxfox%2Fchimera-mk2-core%3Apull%2Cpush for ____:0, 200 OK in 263.2ms @ container/container.go:137(container.Authenticate)
2022/05/18 20:49:56 [628531c4-4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:39e7b07e7410ac8be4d4d7e7de8a2cf0a214278d15410d6eac5828595ab02085 for ____:0, 200 OK in 51.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 404 Not Found in 56.6ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 404 Not Found in 62.9ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034 for ____:0, 200 OK in 75.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-6] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 404 Not Found in 48.2ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-5] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:659bbfc26a5a93b8010ea6b575f7f0a60cdfe6a80e5e154fde755479d301f0ce for ____:0, 200 OK in 58.6ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-8] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 140.3ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:56 [628531c4-7] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 249.2ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:56 [628531c4-9] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:02909e51564bd118b45605cf034515fbad616eeb4547d32e8e43c30ce73a507d for ____:0, 200 OK in 258.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-10] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 305.9ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:56 [628531c4-11] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for ____:0, 200 OK in 434.9ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-16] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for ____:0, 200 OK in 23.1ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:56 [628531c4-14] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:82a003d50dff92ab1725d548b14447af15583ee2f475dfb83cb7582b0026f416 for ____:0, 404 Not Found in 401.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:57 [628531c4-17] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:70dfc71c1e12e4d8a617313a286212d2b3cd0751c3df8c57e9408048d060d956 for ____:0, 200 OK in 360.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:57 [628531c5] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 285.3ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:57 [628531c5-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d5fd17ec1767521cf97f61568096bfc9a7cd9c2d149576a7b43930c5a97062b0 for ____:0, 200 OK in 23.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:57 [628531c4-13] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/k9h6area7tbavls4fewkuetvy?digest=sha256%3A06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 201 Created in 1081.7ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:57 ...es/container/blob.go:56:func1() [E] [628531c4-12] Error inserting package: Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:57 ...es/container/blob.go:56:func1() [E] [628531c4-15] Error inserting package: Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:57 [628531c4-15] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/ro1rkcdp1ib4awzbummsl24oc?digest=sha256%3A09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 500 Internal Server Error in 1077.3ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:57 [628531c4-12] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/dl0e8ctrfrwh1jiu7i6a403y1?digest=sha256%3Afe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 500 Internal Server Error in 1256.4ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:57 ...ntainer/container.go:84:apiError() [E] [628531c4-15] Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:57 ...ntainer/container.go:84:apiError() [E] [628531c4-12] Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/18 20:49:58 [628531c6] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 404 Not Found in 57.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:58 [628531c6-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 404 Not Found in 88.8ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:49:59 [628531c6-3] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 66.6ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:59 [628531c6-4] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ____:0, 202 Accepted in 51.5ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/18 20:49:59 [628531c7] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/95o8db7u0w9snfrt7itn8h5vh?digest=sha256%3A09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 201 Created in 332.7ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:59 [628531c7-2] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/acdmgf4ph173eyp5jcqrgllqk?digest=sha256%3Afe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 201 Created in 390.7ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:49:59 [628531c5-3] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/ab3ohnddvqlltk8ddwffmxgd4?digest=sha256%3A82a003d50dff92ab1725d548b14447af15583ee2f475dfb83cb7582b0026f416 for ____:0, 201 Created in 2063.3ms @ container/container.go:264(container.EndUploadBlob)
2022/05/18 20:50:00 [628531c8] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:6c8411bdc95928eb58cceca5a543cd0c9137383f48b0d3fa03b41afee57548a6 for ____:0, 404 Not Found in 21.5ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:00 [628531c8-2] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/sha256:6c8411bdc95928eb58cceca5a543cd0c9137383f48b0d3fa03b41afee57548a6 for ____:0, 201 Created in 380.9ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:00 [628531c8-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:d9ef2c79f420f4b83129d006441d00286e3f1a696266b4ef6e97c2017c898662 for ____:0, 404 Not Found in 20.9ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:01 [628531c8-4] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/sha256:d9ef2c79f420f4b83129d006441d00286e3f1a696266b4ef6e97c2017c898662 for ____:0, 201 Created in 364.4ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:01 [628531c9] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/testing-2-c69e9a8f0a for ____:0, 404 Not Found in 19.7ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:01 [628531c9-2] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/testing-2-c69e9a8f0a for ____:0, 201 Created in 111.9ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:01 [628531c9-4] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:82a003d50dff92ab1725d548b14447af15583ee2f475dfb83cb7582b0026f416 for ____:0, 200 OK in 77.3ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-5] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d4ba87bb7858f0dd4a60003f011338f3a58b87d0add985652856007fe5ca5034 for ____:0, 200 OK in 86.0ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:02909e51564bd118b45605cf034515fbad616eeb4547d32e8e43c30ce73a507d for ____:0, 200 OK in 100.8ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-6] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:06ecd50008383f7aac0acd7bb5c8c19dcd0d29df93b12871218dd5cec426d6bd for ____:0, 200 OK in 101.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-7] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:70dfc71c1e12e4d8a617313a286212d2b3cd0751c3df8c57e9408048d060d956 for ____:0, 200 OK in 96.9ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-8] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:09d6f9eaf28ac33c6e44a860dbefbcbe157ff4a793effb9a7824f6b453a009cd for ____:0, 200 OK in 77.2ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-10] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:fe18631e174dd7b4a8a9da2f7a7f1f3fb8f40a658bda2c3934612b08d01a1aac for ____:0, 200 OK in 66.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-9] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:659bbfc26a5a93b8010ea6b575f7f0a60cdfe6a80e5e154fde755479d301f0ce for ____:0, 200 OK in 81.1ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-12] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for ____:0, 200 OK in 40.2ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-11] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 for ____:0, 200 OK in 91.5ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-13] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d5fd17ec1767521cf97f61568096bfc9a7cd9c2d149576a7b43930c5a97062b0 for ____:0, 200 OK in 34.7ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-14] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:39e7b07e7410ac8be4d4d7e7de8a2cf0a214278d15410d6eac5828595ab02085 for ____:0, 200 OK in 36.8ms @ container/container.go:339(container.HeadBlob)
2022/05/18 20:50:01 [628531c9-15] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:d9ef2c79f420f4b83129d006441d00286e3f1a696266b4ef6e97c2017c898662 for ____:0, 200 OK in 28.4ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:01 [628531c9-16] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/sha256:6c8411bdc95928eb58cceca5a543cd0c9137383f48b0d3fa03b41afee57548a6 for ____:0, 200 OK in 22.7ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:02 [628531ca] router: completed HEAD /v2/mxfox/chimera-mk2-core/manifests/testing for ____:0, 200 OK in 21.8ms @ container/container.go:477(container.HeadManifest)
2022/05/18 20:50:02 [628531ca-2] router: completed PUT /v2/mxfox/chimera-mk2-core/manifests/testing for ____:0, 201 Created in 295.6ms @ container/container.go:407(container.UploadManifest)
2022/05/18 20:50:06 [628531ce] router: completed POST /api/v1/repos/Chimera.Mark2/core/statuses/c69e9a8f0a5e35a40d292466236c12740c9db5e2 for ____:0, 201 Created in 345.8ms @ repo/status.go:21(repo.NewCommitStatus)

Could you enable dev mode in app.ini so that more detail could be displayed.

[wxiaoguang]

The problem is caused by concurrency requests to database.

In TryInsertPackage/GetOrInsertVersion: Get -> Insert.

Then insert can trigger Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s' error.

[PavelMXFox]

Could you enable dev mode in app.ini so that more detail could be displayed.

Ok

[PavelMXFox]

Hi! With mode=dev.

This appears every time while building cross-platform images with docker buildx. In this build i can't find if any layes drop yet.

Builder log:

#13 4.946 error: failed commit on ref "config-sha256:cf6b0ada9237ffe1fc7a102a6bb21b71034125cb7b2b8d1eeaad8401b8f235ce": unexpected status: 500 Internal Server Error
#13 4.946 retrying in 1s
#13 4.981 error: failed commit on ref "layer-sha256:df4525ed1afc13237ff175f81e53b89e96bbc171bc69940015055e8950362733": unexpected status: 500 Internal Server Error
#13 4.981 retrying in 1s
#13 pushing layers 6.1s done
#13 pushing manifest for mxfox.ru/mxfox/chimera-mk2-core:testing-13-c163e1fa67@sha256:6eda7fb374665a5af6e0420cfa00b017eed96e90a16b17a6aa229abceb137a18
#13 pushing manifest for mxfox.ru/mxfox/chimera-mk2-core:testing-13-c163e1fa67@sha256:6eda7fb374665a5af6e0420cfa00b017eed96e90a16b17a6aa229abceb137a18 1.5s done

Gitea log:

2022/05/21 19:30:48 [628913b7-9] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:2ff2e940e73e30a29749f6cdec49203aa1d1a94b30b59988ce6288e14b836fcd for ******:0, 200 OK in 322.6ms @ container/container.go:339(container.HeadBlob)
2022/05/21 19:30:48 [628913b7-6] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ******:0, 202 Accepted in 533.4ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/21 19:30:48 [628913b8] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:d5fd17ec1767521cf97f61568096bfc9a7cd9c2d149576a7b43930c5a97062b0 for ******:0, 200 OK in 22.7ms @ container/container.go:339(container.HeadBlob)
2022/05/21 19:30:48 [628913b8-3] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:c48a9434d528936656b47c42a842d7f02e61faad1b171854f6d039e8b75fac58 for ******:0, 404 Not Found in 19.8ms @ container/container.go:339(container.HeadBlob)
2022/05/21 19:30:48 [628913b8-4] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ******:0, 202 Accepted in 134.6ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/21 19:30:49 [628913b7-10] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/zetdy8kdtulde2thozhbrr8bj?digest=sha256%3A08cf0fa824af0bf647cefc1557e94b9d375a5c8afc6d14e30e6b813d0d2249fe for ******:0, 201 Created in 1302.0ms @ container/container.go:264(container.EndUploadBlob)
2022/05/21 19:30:49 ...es/container/blob.go:56:func1() [E] [628913b7-11] Error inserting package: Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/21 19:30:49 ...es/container/blob.go:56:func1() [E] [628913b8-2] Error inserting package: Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/21 19:30:49 ...ntainer/container.go:84:apiError() [E] [628913b7-11] Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/21 19:30:49 [628913b7-11] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/sp450bclryqfnguivtgcdkja9?digest=sha256%3Acf6b0ada9237ffe1fc7a102a6bb21b71034125cb7b2b8d1eeaad8401b8f235ce for ******:0, 500 Internal Server Error in 1407.4ms @ container/container.go:264(container.EndUploadBlob)
2022/05/21 19:30:49 [628913b9-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:39e7b07e7410ac8be4d4d7e7de8a2cf0a214278d15410d6eac5828595ab02085 for ******:0, 200 OK in 75.5ms @ container/container.go:339(container.HeadBlob)
2022/05/21 19:30:49 ...ntainer/container.go:84:apiError() [E] [628913b8-2] Error 1062: Duplicate entry '10-_upload' for key 'UQE_package_version_s'
2022/05/21 19:30:49 [628913b8-2] router: completed PUT /v2/mxfox/chimera-mk2-core/blobs/uploads/8lmlkch8ui3qjqobu7j188i6f?digest=sha256%3Adf4525ed1afc13237ff175f81e53b89e96bbc171bc69940015055e8950362733 for ******:0, 500 Internal Server Error in 1080.2ms @ container/container.go:264(container.EndUploadBlob)
2022/05/21 19:30:50 [628913ba] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:cf6b0ada9237ffe1fc7a102a6bb21b71034125cb7b2b8d1eeaad8401b8f235ce for ******:0, 404 Not Found in 43.8ms @ container/container.go:339(container.HeadBlob)
2022/05/21 19:30:50 [628913ba-2] router: completed HEAD /v2/mxfox/chimera-mk2-core/blobs/sha256:df4525ed1afc13237ff175f81e53b89e96bbc171bc69940015055e8950362733 for ******:0, 404 Not Found in 39.6ms @ container/container.go:339(container.HeadBlob)
2022/05/21 19:30:50 [628913ba-4] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ******:0, 202 Accepted in 67.0ms @ container/container.go:157(container.InitiateUploadBlob)
2022/05/21 19:30:50 [628913ba-3] router: completed POST /v2/mxfox/chimera-mk2-core/blobs/uploads/ for ******:0, 202 Accepted in 122.4ms @ container/container.go:157(container.InitiateUploadBlob)

[ChromoX]

I'm experiencing this problem on v1.17.0-rc1, running on Kubernetes.

I have this symptom: #19586 (comment)

[markrexwinkel]

I also experienced this issue with v1.17.0-rc1. When configuring Docker or Podman to use just one thread it works fine, as soon as you configure more parallel threads to push the container layers, error 500 appears. So it seems like a concurrency issue.