Incompatibility in gitbucket migration

[jam7]

Description

Thank you for development. Thank you for implementing migration from gitbucket (#16767). It helps me a lot. However, it doesn't work if a gitbucket repository has many issues or PRs. I inspect it and find the source of problem, so I'm posting this bug report.

While I'm checking the log of migration, I notice there are several logs like Request get issues 49/1, but in fact get 39. This is the source of gitbucket migration problem caused by following incompatibility

• Gitea treat it, fewer issues than perPage issues, as a mark of end of information.
• Gitbucket doesn't return full 49 issues if there are missing issues in the situation like #1 issue, #2 PR, #3 issue.

Easiest way to patch this is to change the migration mechanism. For example, it is possible to change the mechanism to perform migration until an empty array of issues is returned from GetIssue() routine.

+++ b/services/migrations/migrate.go
@@ -331,6 +331,10 @@ func migrateRepository(ctx context.Context, doer *user_model.User, downloader ba

                for i := 1; ; i++ {
                        issues, isEnd, err := downloader.GetIssues(i, issueBatchSize)
+                       isEnd = false
+                       if len(issues) == 0 {
+                               break;
+                       }
                        if err != nil {
                                if !base.IsErrNotSupported(err) {
                                        return err

This works fine while the number of missing issues is small. If there are many missing numbers more than 98 in issues like #1 issue, #2 PR, ..., #99 PR, #100 issue, this doesn't work well. I have no idea how to improve the mechanism, so I'm just posting this bug report.

Gitea Version

1.21.10

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

Gitea in docker, gitea/gitea:1.21.10.

Database

None

[lunny]

But how did you know it's end if there is one middle page that contained issues that are all pull requests?

[jam7]

Yes. That is the problem and I have no idea how to improve the mechanism, so I'm just posting this bug report.

[jam7]

Maybe, it's something what gitbucket and other github clone must take care of. But, it's still good to know the reason of this kind of problems... You know. ;-)

[lunny]

Maybe we can rewrite the function GithubDownloaderV3.GetIssues to use this fields(Next Page, Last Page) to confirm whether it's end.

[jam7]

Thank you for suggestion. Unfortunately, gitbucket doesn't support those fields (Next Page, Last Page).

$ http_proxy= HTTP_PROXY= curl -L -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" "http://localhost:50856/api/v3/repos/CWB-LIB/Library/pulls?state=all&sort=created&direction=asc&page=6&per_page=49" -u USER:PASSWD -I
HTTP/1.1 200 OK
Date: Mon, 08 Apr 2024 02:48:31 GMT
Set-Cookie: JSESSIONID=node06o594avhud4rgctaefj2zi1z102.node0; Path=/; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: application/json
Transfer-Encoding: chunked

github.com

$ curl -L -H "Accept: application/vnd.github+json" -H "X-GitHub-Api-Version: 2022-11-28" "http://api.github.com/repos/go-gitea/gitea/issues?state=all&sort=created&direction=asc&page=1&per_page=49" -I
HTTP/1.1 301 Moved Permanently
Content-Length: 0
Location: https://api.github.com/repos/go-gitea/gitea/issues?state=all&sort=created&direction=asc&page=1&per_page=49

HTTP/2 200
server: GitHub.com
date: Mon, 08 Apr 2024 03:03:52 GMT
content-type: application/json; charset=utf-8
cache-control: public, max-age=60, s-maxage=60
vary: Accept, Accept-Encoding, Accept, X-Requested-With
etag: W/"db7bdfe2b0d8646e5d376ddaef072ae2cd1d0160dfdf82de1f39758b3eb8f264"
x-github-media-type: github.v3; format=json
link: <https://api.github.com/repositories/72495579/issues?state=all&sort=created&direction=asc&page=2&per_page=49>; rel="next", <https://api.github.com/repositories/72495579/issues?state=all&sort=created&direction=asc&page=615&per_page=49>; rel="last"
x-github-api-version-selected: 2022-11-28
access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
access-control-allow-origin: *
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'
x-ratelimit-limit: 60
x-ratelimit-remaining: 58
x-ratelimit-reset: 1712548983
x-ratelimit-resource: core
x-ratelimit-used: 2
accept-ranges: bytes
x-github-request-id: D4D4:167ECD:1A8817B:1AA9EFD:66135E98

[lunny]

So maybe gitbucket needs to make some improvements to their API, otherwise, we cannot know whether the pagination is ending.

[jam7]

Just an update. I've read gitbucket source code and noticed it doesn't support per_page at all. I check again the actual message from gitea log. It is 2024/04/09 07:40:41 ...migrations/github.go:434:GetIssues() [T] Request get issues 49/1, but in fact get 25. The internal per_page of gitbucket is just 25. This is the real source of this problem. Sorry for wrong description in previous posts.

So, it may be possible to use 25 as a hard coded per_page value for only gitbucket migration until gitbucket changes it. Is such kind of patch acceptable for gitea? If it's acceptable, I'll try to create a PR.

[lunny]

Just an update. I've read gitbucket source code and noticed it doesn't support per_page at all. I check again the actual message from gitea log. It is 2024/04/09 07:40:41 ...migrations/github.go:434:GetIssues() [T] Request get issues 49/1, but in fact get 25. The internal per_page of gitbucket is just 25. This is the real source of this problem. Sorry for wrong description in previous posts.

So, it may be possible to use 25 as a hard coded per_page value for only gitbucket migration until gitbucket changes it. Is such kind of patch acceptable for gitea? If it's acceptable, I'll try to create a PR.

Thank you for the investigation. I think it's acceptable. Please send a PR.