Slow repository browsing in 1.14.x

[tsowa]

• Gitea version (or commit ref): 1.14.x
• Git version: 2.31.1
• Operating system: FreeBSD 13
• Gitea built using ports collection (www/gitea)
• Gitea started by startup script provided by www/gitea port
• Database (use [x]):
  • PostgreSQL 12.6
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (https://try.gitea.io/tsowa/FreeBSD_ports)
  • No
• Log gist: https://www.ttmath.org/gitea.log

Description

I saw a similar thread but there is "windows" in the title so I create a new issue. Gitea 1.14.x is much slower in repository browsing than Gitea 1.13.

Sample repo running with 1.14.1:
https://gitea.ttmath.org/FreeBSD/ports
Try to open any directory, for example:
https://gitea.ttmath.org/FreeBSD/ports/src/branch/main/audio
It takes between 50-150 seconds to open a page.

The same repo running with 1.13.7:
https://giteaold.ttmath.org/FreeBSD/ports
Try to open similar directory, for example:
https://giteaold.ttmath.org/FreeBSD/ports/src/branch/main/audio
I takes about 5 seconds.

You can see the same problem on try.gitea.io:
https://try.gitea.io/tsowa/FreeBSD_ports
But you have a cache so you have to find a directory which was not open before. Opening such a page takes 100-300 seconds.

Let me know if more info is needed.

[zeripath]

This is because the algorithm was changed in 1.14 due to a problem with go-git causing significant memory issues. Thank you for the test cases though because they will provide tests to improve the current algorithm.

If you are suffering significant slow downs here you can switch back to the gogit build by adding gogit to your TAGS during building.

We would otherwise appreciate help in improving the performance of the algorithm for the pure git version.

[tsowa]

Thanks for the hint with TAGS. I don't have time to make more tests now but I found something interesting.

When browsing my repository with gitea I see in htop following git processes:

22304 root       20   0 12876  2100 S  0.0  0.0  0:00.00 daemon: /usr/bin/env[22305]
22305 git2       31   0  926M  254M S 136.  0.8  1:29.80 └─ /usr/local/sbin/gitea web
22839 git2       21   0  952M  158M S  3.3  0.5  0:01.11    ├─ /usr/local/bin/git -c credential.helper= -c protocol.version=2 rev-list --format=%T 9ea557779ce520c206f223f6f7b48fcc52f92dad
22840 git2       27   0 1103M  275M S 13.5  0.8  0:04.59    └─ /usr/local/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch

These processes were running for about one minute so I have run the first git process by hand:

$ cd /var/db/gitea2/gitea-repositories/freebsd/ports.git
$ /usr/local/bin/git -c credential.helper= -c protocol.version=2 rev-list --format=%T 9ea557779ce520c206f223f6f7b48fcc52f92dad | wc -l

and it gave me 1087346 rows. I suppose the millions rows are then pass to the second git process.

I have piped output from the first git to the other:

$ /usr/local/bin/git -c credential.helper= -c protocol.version=2 rev-list --format=%T 9ea557779ce520c206f223f6f7b48fcc52f92dad | /usr/local/bin/git -c credential.helper= -c protocol.version=2 cat-file --batch > swinka.txt

it takes about 15 seconds and shows that file swinka.txt is larger than 1 GB

$ ll -h swinka.txt 
-rw-r--r--  1 git2  git2   1,4G 10 maj 22:47 swinka.txt

so there is a lot of data to pass between gitea and git. So the question is: is it really needed for the first git process to return one milion rows?

[zeripath]

@tsowa unfortunately yes but it should be relatively fast - the issue will be that the structure of some repos will actually require that million of rows to be checked more than a few times. Determining which commit a file is related to is not a simple task in git - and although there's a commit graph we don't have a good way of querying it.

(It shouldn't take 15s to pipe those two commands together - you're slowing things down by allocating file space - you should pipe the output to null btw.)

There are a few more improvements to that function that can be made - for a start the function is not optimised for our collapsing of of directories containing a single document - and writing a commit graph reader would be part of that.

The gogit backend does have a commitgraph reader but it is not frugal with memory at all. I need to spend some time making a reader that is much more frugal and stream like but I haven't had the time. (See the technical docs https://github.com/git/git/blob/master/Documentation/technical/commit-graph.txt)

In the end though we need to move rendering of last commit info out of repo browsing and in to an ajax call. Again something I haven't had time to do.

[zeripath]

One question - have you disabled the commit cache? If so please re-enable it.

[tsowa]

It was enabled by default but the 'adapter' option was set to 'memory'. Now I have installed memcached and changed adapter to 'memcache' and a difference is visible.

Opening https://gitea.ttmath.org/FreeBSD/ports for the first time took 79766ms and for the second time only 3063ms. Opening https://gitea.ttmath.org/FreeBSD/ports/src/branch/main/audio for the first time 141221ms and later 37205ms.

But I see that you are calling a lot of git processes, I have created a small git wrapper in such a way:

#include <unistd.h>
#include <fstream>
#include <iostream>

int main(int argc, char * argv[], char * envp[])
{
    std::ofstream file("/home/tomek/git.log", std::ios_base::out | std::ios_base::app);

    if( file )
    {
        file << "git ";

        for(size_t i=0 ; argv[i] ; ++i)
        {
            file << argv[i] << " ";
        }

        file << std::endl;
        file.close();
    }

    return execve("/usr/local/bin/git.org", argv, envp);
}

I have moved original /usr/local/bin/git to /usr/local/bin/git.org and have compiled above program as /usr/local/bin/git. And it gives me git.log with all git operations and I see that sometimes you are calling the git binary 300 times in one request:

cat ~/git.log | wc -l
     335

So it cannot be fast, this remains me of the old days when we were using cgi scripts. Is there a reason you are calling git directly instead of using a git library such as libgit2?

[lunny]

Could you also count what's the git command Gitea invoked in these 335 commonds?

This is because when browsing, Gitea will get last commit message for every dir/file on the ui. For v1.13.0, we use go-git which is a pure go git library, for v1.14.x, we have two versions for windows because the library have some memory problems. And maybe you could try to compile the go git version yourself to check what's the different between them.

[fnetX]

Hello there, looking at this from Codeberg's perspective (issue)

As you can see, we're also suffering from the slow repository browsing which affects the overall performance of our machine. While setting up a Redis cache works well for us, we would like to improve the initial generation on cache misses, too.
Today, we tried to find some more information about the bottleneck, I hope this is useful for you:

We suspect especially this command where each folder is checked for the latest commit, executing /usr/bin/git -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --format=%T <commit> which has a terrible performance (loading the entire commit history).

The idea makes sense to us: getting all commits, checking if the folder or file was touched. But the logic without gogit doesn't appear to stop after all necessary information was loaded, but rather continue serving up the entire history, even if all files or subfolders in a folder have already been hit by a recent commit.
While we didn't completely understand the gogit logic yet, it appears to be a little smarter at this point and only looking so far back in history as necessary to retrieve the information it's looking for.

We assume that the process should be stopped before it ends if all information was provided.

It looks like there's a lot of stuff to be improved with the native git backend, and some actions will probably always be slower because they cannot directly interface with git operations (e. g. directly working on git results while they are fetched instead of the piped input). It might a good idea to turn back to gogit for all systems when the memory issues are resolved, or, look for another git library (that is more native and maybe faster than gogit), but offers some better interface (go-bindings for libgit2?)

Please let us know if we can provide further assistance in improving this performance issue.

Some other random observations that might be interesting to you:

• multiple requests for the same resource try to generate it concurrently on cache miss, the operation doesn't get queued (thus, requesting the same page twice results in generating it twice until the cache is filled)
• git processes aren't stopped when the requesting TCP connection is closed
  • if a connection times out (proxy), the process is still running in the backend
  • a user reloading the page easily doubles the resources for this operation
  • it's possible to DoS a huge server by simply spamming F5 (reload) on a page with a cache miss with minimum cost at the attackers side (no need to keep the connection open)
• initial generation with this method will always be very slow if some files (e.g. a README , LICENCE, .gitignore, gitattributes, dockerignore etc) weren't touched for a long time
• pushing to a repo invalidates the full redis cache, even if only parts of the information changed (e.g. a subfolder was updated), thus active repos won't profit very much from the cache
• timed out git commands (by adjusting the timeout in Gitea) keeps them listed as running in the Gitea admin monitoring section, although the processes don't exist in the system any more

[zeripath]

Take a look at #15891

[zeripath]

@fnetX thanks for your long comment.

---

It's worth remembering that the issue precipitating the pure git backend was memory use. I've submitted a patch to go-git which should cause much lower memory load. Until that is in and working correctly, go-git will happily load in huge blobs in to memory - storing them in caches even when you want to check the size of the object. It's really worth being clear that that is an absolutely intolerable situation.

Further the issues you are highlighting in the last section are not new to the native git backend. They're present in a different way in the go-git backend just in a way you can't track. I've long advocated for changing to a more gitlab approach for this and/or for passing in request contexts to terminate things - I'm really happy to work on this - but I haven't had a chance to do this - and to be honest none of you are paying me.

---

Now going on to the get last commit algorithm.

All algorithms have a balance between memory and time. The current algorithm is highly optimised against memory use. If we are happy to use more memory that can be improved.

We suspect especially this command where each folder is checked for the latest commit, executing /usr/bin/git -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-list --format=%T <commit> which has a terrible performance (loading the entire commit history).

The idea makes sense to us: getting all commits, checking if the folder or file was touched. But the logic without gogit doesn't appear to stop after all necessary information was loaded, but rather continue serving up the entire history, even if all files or subfolders in a folder have already been hit by a recent commit.
While we didn't completely understand the gogit logic yet, it appears to be a little smarter at this point and only looking so far back in history as necessary to retrieve the information it's looking for.

Looking at the length of time the rev-list process is running a bit of a distractor. Yes the go-git process can stop once it's finished looking at all the parents and the paths, but it's a question of memory and time spent tracking the parents. git rev-list avoided tracking those parents - and grabbing the root tree saved a lot of time - but we could add in %P to the format to allow tracking of parents and could allow termination once all appropriate parents are determined - but I don't think it's the primary cause of delays.

The greatest speed up in #15891 is actually preemptive passing the next tree ID to the git cat-file process as soon as we know what it's going to be. A large proportion of time appears to be spent waiting for go to fill the read buffer from the other process. This is where the go-git algorithm can be quicker as it avoids that by reading files in directly.

Some other random observations that might be interesting to you:

• multiple requests for the same resource try to generate it concurrently on cache miss, the operation doesn't get queued (thus, requesting the same page twice results in generating it twice until the cache is filled)

• git processes aren't stopped when the requesting TCP connection is closed

  • if a connection times out (proxy), the process is still running in the backend
  • a user reloading the page easily doubles the resources for this operation
  • it's possible to DoS a huge server by simply spamming F5 (reload) on a page with a cache miss with minimum cost at the attackers side (no need to keep the connection open)

• initial generation with this method will always be very slow if some files (e.g. a README , LICENCE, .gitignore, gitattributes, dockerignore etc) weren't touched for a long time

• pushing to a repo invalidates the full redis cache, even if only parts of the information changed (e.g. a subfolder was updated), thus active repos won't profit very much from the cache

• timed out git commands (by adjusting the timeout in Gitea) keeps them listed as running in the Gitea admin monitoring section, although the processes don't exist in the system any more

These are all longstanding issues and I am aware of them. I would love to spend time fixing these but I am limited in my time and availability.

---

Honestly I wish you'd just talked to me directly. I'm always on Discord and could have told you and kept you abreast of what was going on and my progress in trying to speed this up.

[zeripath]

oh my - I think I know how to seriously improve this. I think I've been way too distracted by the way it was done in the go-git implementation and there's genuinely a much quicker way to do this.

[fnetX]

Hey, thank you very much for the explanation.

I've submitted a patch to go-git which should cause much lower memory load. Until that is in and working correctly ...

I somehow thought this was already in and just needed some further improvements, my bad.

and to be honest none of you are paying me.

Yes, we can mainly offer to being thankful as long as we aren't paid for anything either. ❤️
But let's see if we can figure something out.

Honestly I wish you'd just talked to me directly. I'm always on Discord

Yeah, the others told me that, too. But since Discord is a proprietary app that kept crashing my computer back when I last used it, I decided against this and went for dumping our findings somewhere, hoping they are of any use. Chose this issue over the thread on Codeberg as it seemed to better fit in this topic here.

I think I know how to seriously improve this

Sounds like good news. Please tell us if there's anything we can do.

[zeripath]

oh my - I think I know how to seriously improve this. I think I've been way too distracted by the way it was done in the go-git implementation and there's genuinely a much quicker way to do this.

Unfortunately this doesn't work.

The idea was to use git log --format=%H --raw -t --no-abbrev --reverse COMMIT_ID -- paths but I can't come up with a way to stop it from listing the contents of the trees - meaning that it takes even longer.

If I could figure out a way to not list the contents of the trees this would be definitely faster than the go-git version.

[fnetX]

What about adding -n 1? Correct me if I am completely mistaken, because I neither fully understand the Gitea backend yet, nor do I know how git works, but this seems to give you the latest commit of a path and gives the same result as Gitea currently gives.