Huge pull request creation timeout

[TitaniumHocker]

Description

I have huge pull request with more that 1kk diff lines and it always fails with timeout.
I raised the timeout on the reverse proxy side to 5 minutes, and then to 10 minutes, but this did not help. I don’t know if there is any point in raising the timeout anymore.

Here is some errors from logs:

Nov 08 17:04:46 gitea-hostname gitea[699]: 2023/11/08 17:04:46 ...ll/patch_unmerged.go:164:unmergedFiles() [E] Unable to run ls-files -u -z! Error: git ls-files -u -z: context deadline exceeded
Nov 08 17:04:46 gitea-hostname gitea[699]: 2023/11/08 17:04:46 ...ers/web/repo/pull.go:1297:CompareAndPullRequestPost() [E] NewPullRequest: signal: killed

I tried to add more RAM and CPU to VM, but it doesn't help. Judging by htop, this operation does not waste a lot of resources.

Gitea Version

1.20.5

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

2.30.2

Operating System

Debian 11

How are you running Gitea?

Selfhosted with systemd unit, downloaded from releases page on github.
VM instance has 8GB of RAM and 4 vCPU.

Database

PostgreSQL

[KN4CK3R]

You may need to adjust these timeouts:

gitea/custom/conf/app.example.ini

Lines 700 to 707 in 69d98f8

	;; Git Operation timeout in seconds
	;[git.timeout]
	;DEFAULT = 360
	;MIGRATE = 600
	;MIRROR = 300
	;CLONE = 300
	;PULL = 300
	;GC = 60

[TitaniumHocker]

You may need to adjust these timeouts:

I don't see timeout for git ls-files in this list. Besides, my operation timeouts on reverse proxy side. I don't want to increase timeouts - 10 minutes alreary too much.

[KN4CK3R]

git ls-files uses the DEFAULT timeout setting.

[TitaniumHocker]

git ls-files uses the DEFAULT timeout setting.

Seems like it doesn't work, process fails on reverse proxy timeout in 10 minutes, not in 6 minutes with default value.

[lschwahn]

Hi,

we are observing a similar behavior:

In our case it's not, that the pull request itself is that big. But the amount of files identified by following Function ist ~ 1000.

gitea/services/pull/patch_unmerged.go

Line 57 in 709a376

func readUnmergedLsFileLines(ctx context.Context, tmpBasePath string, outputChan chan *lsFileLine) {

Because of this the following loop runs several minutes, resulting in a timeout because of our Reverse-Proxy (which was set to
60 secs).

gitea/services/pull/patch_unmerged.go

Line 161 in 709a376

for line := range lsFileLineChan {

We set the reverse-proxy timeout to 6 Minutes, to match the gitea default timeout values. As a workaround this "resolved" the
Proxy-Timeout-Errors. But the time spent to create a pull request is still not satisfying.

Gitea Version
1.21.0

Git Version
2.40

Operating System
Rocky-Linux 8

How are you running Gitea?
Docker. We reproduced the behavior on three different machines with Rocky Linux 8 installed:

PowerEdge R630: CPU: 2 * Intel Xeon E5-2630 v3 (2.4GHz; 8 Cores), Storage: SAS, Memory: 128GB
PowerEdge R740: CPU: 2* Intel Xeon Gold 6126 (2.6GHz; 12 Cores), Storage: NVMe, Memory: 768GB
Virtual Server (Hyper-V): CPU: 4 Cores, Memory: 8GB

Database
MariaDB

[achepukov]

As a workaround you can create PR based on branch equals to base branch, and then push new commits into it.

[bartvdbraak]

We're also running into this issue, unfortunately.

Even with:

[git.timeout]
DEFAULT = 600

[brechtvl]

It appears that running git update-index --remove for every individual file is slow. Batching them all together is much faster.

I have a change here that helps for file removal, but not file addition yet. I won't have time to finish this, but maybe someone else will pick it up.
https://github.com/blender/gitea/commits/fix-slow-check-many-files/