Performance issues with conversation tab in PR with comments/threads and images/rst files changes

[Persimmonboy]

Description

Hello Team,

We are experiencing significant slow loading times (about 3 minutes) in Gitea when dealing with pull requests (/docs/application-performance-management/pulls/6) that include about 580 review sections with 700 comments and file changes, with about 75 files involved. This problem occurs in Gitea version 1.19.3 and persists after upgrading to version 1.21.6, deployed on Kubernetes with Redis and an external PostgreSQL database. We've confirmed there are no issues with disk, CPU, or RAM usage, as well as with the database itself.

The pull request page, especially the "Conversation" tab, loads very slowly. However, if we recreate the pull request for the same branches and add a similar number of comments, using a simple word like "test" in each, the loading time is normal. This suggests that the slowdown may be related specifically to the opening of issue threads linked to certain parts of the code. The other tabs in the pull request, such as "Commits" and "Files Changes", load within a reasonable time.

It's important to note that there are several pull requests experiencing this issue, which suggests a consistent pattern or underlying problem.

Logs and Stacktrace outputs are in attachments:

gitea-logs.log
gitea-diagnosis-20240618-103535.zip

Could this issue be indicative of a bug, or is there a potential optimization strategy that we might consider? Any advice or assistance in resolving these slowdowns would be greatly appreciated.

Gitea Version

1.21.6

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

2.40.1

Operating System

Linux gitea-85d5dc6896-6brvw 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 Linux

How are you running Gitea?

We are running Gitea on Kubernetes, deployed using the official Helm chart available at https://gitea.com/gitea/helm-chart. This setup includes Redis for caching and an external PostgreSQL database for data storage.

Database

PostgreSQL

[Persimmonboy]

I think the problem is related to the frontend part on the client side that handles the review sections with comments, as the backend seems to load in ~15 seconds, after which about 3 minutes or more comments are loaded:

[wxiaoguang]

Since 1.21 is outdated and won't get regular fixes, could you upgrade to 1.22-nighty (it will be 1.22.1 soon and contains a lot of bug fixes)?

Then could you provide the browser's profile?

Update: if it's impossible to upgrade, continue debugging with 1.21 might also be OK .....

[wxiaoguang]

Well, indeed ..... slow $.fn.tab

[Persimmonboy]

@wxiaoguang I updated to version 1.22-nightly, but nothing changed. PR conversation tab loading remained slow.

Current version:

gitea-69c59cbf5f-d6nd2:/var/lib/gitea$ gitea --version
Gitea version 1.22.0+56-ge8e43a7ee4 built with GNU Make 4.4.1, go1.22.4 : bindata, timetzdata, sqlite, sqlite_unlock_notify
gitea-69c59cbf5f-d6nd2:/var/lib/gitea$

Link to bucket with performance profle json

[wxiaoguang]

Thank you very much. I will try to reproduce it and try to fix

[wxiaoguang]

I can reproduce it now. WIP: Fix tab performance #31437 .

Need to figure out how to fix ....

[Persimmonboy]

@wxiaoguang

Thanks for the quick reply!

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

I'm trying to reproduce the problem, and perfomance profile looks similar, but it loads faster, because in my problematic PR resolved reviews with comments and the comments themselves are more, which I mentioned in the first post, than in the test one.

Here is my test repository:

https://demo.gitea.com/ashakhba/application-perfomance-managment/pulls/1

[wxiaoguang]

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

That's another point, so maybe also related.

[wxiaoguang]

Link to bucket with performance profle json

According to this profile report, your instance's slowdown is 100% related to the $.fn.tab

Do you think it's not a combination of comments with closed reviews that can also have comments and just comments?

However, it could still be a problem (another problem ......)

---

Are you able to run main-nightly (aka unreleased 1.23-nightly) in your production? If yes, I could provide more changes in the main branch, because some refactoring changes might not be suitable to backport to the stable 1.22 release.

If you could only use 1.22.x at the moment .... I could try to backport some changes if they are proven to work in 1.23, but I can't 100% guarantee at the moment.

[Persimmonboy]

@wxiaoguang

This is the dev environment I reproduced using helm chart gitea from the prod gitea data and database, so I can update.

The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right?

[wxiaoguang]

The gitea/gitea:nightly-rootless from dockerhub is required version 1.23-nightly right?

Yes, nightly-rootless is for unreleased 1.23-nightly, but I haven't made any attempt yet 🤣 , I will notify here when I would have made some progress.