Pull Requests listing is way slower on Windows

[ramasoft-dv]

• Gitea version (or commit ref): 1.14
• Git version: 2.26.2
• Operating system: Windows Server 2016 Standard
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No

In 1.13.6 it was fast when going in the Pull Requests tab.
We've migrated to 1.14 today (2024-04-11) and now it's 20x slower.
We have "ISSUE_PAGING_NUM" set to 50. And it takes 30 seconds to load the pull requests in 1.14. In 1.13.6, it took less than 1 second.
I put back the default value to 10, it's better but it still takes 5-6 seconds. I think there is a missing index or something not retro-compatible.

How can I help you to investigate the issue ?

Kind regards.

[ramasoft-dv]

It does not seems to be related to the DB. I've tried to move gitea service in another faster computer. The pull requests listing was 2x faster : 30 seconds => 15 seconds.
Moreoever, I've set SQL.Logs to true to check which query could take too much time, and actually none takes more than 1-2 ms.
So i'm pretty sure it's not comming from DB, but more from something new done after having retrieved the pull requests, while compiling informations.
Also: Accessing to individual pull request, is as fast as in 1.13.6.
So the issue come from only when accessing to Pull Requests Tab Page.

[lunny]

Could you enable pprof and paste the profile file

[ramasoft-dv]

Yes, here you are:

profile.log

[lunny]

Have you load the pull request list page when you enable pprof? It seems there is no any functions about that.

[ramasoft-dv]

Yes I did, but if it didn't worked, then I can make it again, no problem. Here you are:
profile.log

[ramasoft-dv]

This is another profile.log:

1. I started gitea service : gitea.exe
2. I logged in with my user
3. I went on the specific repository (Ramasoft.NET)
4. I clicked on Pull Requests Tab
5. I went on http://localhost:6060/debug/pprof/, clicked on "profile" which downloaded me the following file:
   profile.log

And this is what I can see in the page loading times:

[ramasoft-dv]

And this is our app.ini file (a little bit modified to remove sensible info)
app.txt

[lunny]

Could you also paste your gitea logs?

[ramasoft-dv]

Yes, of course. Here you are:

1. I changed the app.ini I provided you just earlier by defining the level of Logs to Trace
2. I started gitea service : gitea.exe
3. I logged in with my user
4. I went on the specific repository (Ramasoft.NET)
5. I clicked on Pull Requests Tab
6. I went on http://localhost:6060/debug/pprof/ and saved everything that was in this page.
7. Killed gitea service

These are the files:

1. Logs: log.txt
2. pprof files:
   block.txt
   cmdline.txt
   fullGoRoutineStackDump.txt
   goroutine.txt
   heap.txt
   mutex.txt
   profile.log
   threadcreate.txt
   trace.log
   allocs.txt

And this is how much time it took to go to Pull Request Tab:

Kind regards,
David.

[zeripath]

Thanks for extensive logging files and pprof files but please could you read https://docs.gitea.io/en-us/logging-configuration/#debugging-problems

STACKTRACE_LEVEL should generally be left unconfigured (and hence kept at none). There are only very specific occasions when it useful.

Your logs are essentially unreadable because of this setting. You have not given us the router logs so I can't tell which URL is slow.

Which url exactly is slow?

The pprof files are also in the incorrect format - pprof files should be binary files. You can get one quite easily:

wget -O  30s-cpuprofile.out 'http://localhost:6060/debug/pprof/profile?seconds=30' 

And then request the broken page whilst this is downloading.

The text files aren't really very useful at all.

The profile.log in this #15401 (comment) is in the right format but doesn't actually contain the slow page request so nothing can be inferred.

[ramasoft-dv]

Ok, no prob. I change the logging settings to the one in your documentation: Level=debug, and SQL.Log = False.

Thank you for the explanation of how pprof works. I was using it improperly.

Also for your question this is the url that is slow: http://localhost:54326/Ramasoft/Ramasoft.NET/pulls

This is what i did:

1. Change the app.ini to have the correct log level and other settings more suited for debugging
2. Run gitea service
3. Run your query (wget...)
4. Refresh the Pull Request tab

These are the files I can give to you:

30s-cpuprofile.log
logs.txt

It is ok now ?

Kind regards,
David.

[zeripath]

One quick thought.

Most Windows users have seriously long path environment variables. Windows doesn't hash this path unlike Linux and Mac.

We're mostly using git directly these days.

Set your [git] PATH correctly as this could be a cause of serious slow downs.