Gitea action runner fails to start randomly on low-end hardware

[repo-ooo]

Description

I'm running gitea and gitea runner in docker on low-end hardware (Intel J1900). The actions job is very hard to start. Mostly, the actions run page is stucked on "Set up job" without any output, and finally failed after 10~12 minutes.

I have to repeatedly click cancel and rerun all jobs on the actions run page, and quickly check if the job is successfully sent to runner via docker log gitea-runner. If I'm lucky, it might start successfully on the first try; if not, it could fail more than 20 times in a row.

I believe this problem is due to the low-end hardware. When I'm running git clone https://git.mydomain.com/a/b, It may also fail at the first time, and succeed at the second time. Unfortunately, it's easy to run git clone 2 times, but it's hard to cancel and rerun action jobs again and again.

Is it possible to add some retry policy when starting an action job?

2025/04/14 19:27:15 ...s/process/manager.go:188:Add() [T] Start 67fcf113: POST: /api/actions/runner.v1.RunnerService/FetchTask (request)
2025/04/14 19:27:15 ...eb/routing/logger.go:47:func1() [T] router: started   POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0
2025/04/14 19:27:16 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 127.0.0.1:48944, 200 OK in 2495.5ms @ healthcheck/check.go:67(healthcheck.Check)
2025/04/14 19:27:16 ...s/process/manager.go:231:remove() [T] Done 67fcf111-5: GET: /api/healthz
2025/04/14 19:27:17 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /admini/repo1/actions/runs/26/rerun for 192.168.1.100:0, elapsed 3563.8ms @ actions/view.go:366(actions.Rerun)
2025/04/14 19:27:17 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3520.1ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:18 ...s/process/manager.go:188:Add() [T] Start 67fcf116: POST: /api/actions/runner.v1.RunnerService/FetchTask (request)
2025/04/14 19:27:18 ...eb/routing/logger.go:47:func1() [T] router: started   POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0
2025/04/14 19:27:19 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3494.3ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:19 ...eb/routing/logger.go:102:func1() [I] router: completed POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, 500 Internal Server Error in 5548.4ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf111-4: POST: /api/actions/runner.v1.RunnerService/FetchTask
2025/04/14 19:27:19 ...dels/actions/task.go:246:CreateTaskForRunner() [T] runner labels: [linux_amd64 ubuntu-22.04]
2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin
2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117: git(dir:/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin (from 67fcf111) (normal)
2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf117: git(dir:/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= hash-object --stdin
2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-parse
2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117-2: /usr/bin/git rev-parse [repo_path: /data/git/repositories/admini/repo1.git] (from 67fcf111) (normal)
2025/04/14 19:27:19 ...s/process/manager.go:231:remove() [T] Done 67fcf117-2: /usr/bin/git rev-parse [repo_path: /data/git/repositories/admini/repo1.git]
2025/04/14 19:27:19 ...dules/git/command.go:292:Run() [D] git.Command.RunDir(/data/git/repositories/admini/repo1.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2025/04/14 19:27:19 ...s/process/manager.go:188:Add() [T] Start 67fcf117-3: /usr/bin/git cat-file --batch [repo_path: /data/git/repositories/admini/repo1.git] (modules/git/repo_base_nogogit.go:65) (from 67fcf111) (normal)
2025/04/14 19:27:20 ...git/commit_status.go:480:NewCommitStatus() [D] NewCommitStatus[/data/git/repositories/admini/repo1.git, 4315663afe45a375e15bb0c8ae0ea0379eabf401]: 3
2025/04/14 19:27:20 ...s/process/manager.go:188:Add() [T] Start 67fcf118: POST: /api/actions/runner.v1.RunnerService/FetchTask (request)
2025/04/14 19:27:20 ...eb/routing/logger.go:47:func1() [T] router: started   POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0
2025/04/14 19:27:22 ...eb/routing/logger.go:68:func1() [W] router: slow      POST /api/actions/runner.v1.RunnerService/FetchTask for 192.168.96.3:0, elapsed 3494.8ms @ <autogenerated>:1(http.Handler.ServeHTTP-fm)
2025/04/14 19:27:22 ...ons/commit_status.go:30:CreateCommitStatus() [E] Failed to create commit status for job 26: GetLatestCommitStatus: context canceled

Possible related issues:
#32348, #33492, #33497

Gitea Version

1.23.6

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

No response

Operating System

Synology DSM 7.2.2, kernel version 4.4.302+

How are you running Gitea?

docker-compose up -d

services:
  server:
    image: gitea/gitea:1.23.6
    container_name: gitea
    environment:
      - USER=git
      - USER_UID=1027
      - USER_GID=100
      - GITEA_CUSTOM=/data/gitea
      - DOMAIN=git.mydomain.com
      - SSH_DOMAIN=git.mydomain.com
      - ROOT_URL=https://git.mydomain.com/
    volumes:
      - ./gitea:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "5030:3000"
      - "5022:22"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 30s

  runner:
    image: gitea/act_runner:nightly # v0.2.11+15-gb1ae30d
    container_name: gitea-runner
    depends_on:
      server:
        condition: service_healthy
    environment:
      CONFIG_FILE: /config.yaml
      GITEA_INSTANCE_URL: http://server:3000/
      GITEA_RUNNER_REGISTRATION_TOKEN: <some token>
      GITEA_RUNNER_NAME: host-docker-runner
      GITEA_RUNNER_LABELS: "linux_amd64:docker://node:16.20.2-slim,ubuntu-22.04:docker://gitea/runner-images:ubuntu-22.04-v25.04.01"
    volumes:
      - ./config.yaml:/config.yaml
      - ./runner:/data
      - /var/run/docker.sock:/var/run/docker.sock

The domain git.mydomain.com is resolved to 192.168.0.100, and reverse proxied by Synology DSM to port :5030.

Database

SQLite