always running processes from 1.15.3

[dannyzcy]

Gitea Version

1.15.3

Git Version

2.33.0.windows.2

Operating System

windows

How are you running Gitea?

running gitea by gitea-1.15.3-windows-4.0-amd64.exe

Database

MySQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

always running processes from 1.15.3

the bug from 1.15.3

not appear in 1.15.2

Screenshots

[zeripath]

This is a duplicate report.

Upgrade to 1.15.4 (agh not awake!) - go to the admin monitor page and at the bottom there is a process manager.

That will then tell you what bit of code opened these processes and we'll be able to figure out where the missing close is.

[zeripath]

As this is from the latest version and I can't find the issue number (right now) that this duplicates - I am going to reopen.

@dannyzcy - could you please go to the admin page I mentioned above and show me the processes table at the bottom of the monitor page.

That will allow us to figure out what is opening these processes.

Then we will need to correlate the times with the logs as per https://docs.gitea.io/en-us/logging-configuration/#debugging-problems

[dannyzcy]

The picture above may already be the processes table at the bottom of the monitor page? right?

logs:

2021/09/24 15:42:30 ...ters/private/serv.go:380:ServCommand() [D] Serv Results:
	IsWiki: false
	IsDeployKey: false
	KeyID: 31	KeyName: rsa-key-20210922
	UserName: 1234567
	UserID: 40
	OwnerName: test
	RepoName: test
	RepoID: 179
2021/09/24 15:42:30 ...s/context/context.go:740:1() [D] Session ID: dd55bf8aa898ea07
2021/09/24 15:42:30 ...s/context/context.go:741:1() [D] CSRF Token: cqktVIq6OCIK2gNAMb36lLv4JY86MTYzMjQ2OTM1MDE3MjI2MjUwMA
2021/09/24 15:42:30 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\test\test.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2021/09/24 15:42:30 ...dules/git/command.go:120:RunInDirTimeoutEnvFullPipelineFunc() [D] E:\gitea\gitea-repositories\test\test.git: D:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2021/09/24 15:42:31 modules/ssh/ssh.go:150:publicKeyHandler() [D] Handle Public Key: Fingerprint: SHA256:rcLnaYptHJVEGdltjzt1P0isyRNB9pi02rpJKCc41RA from 10.123.123.140:57680
2021/09/24 15:42:31 modules/ssh/ssh.go:231:publicKeyHandler() [D] Handle Public Key: 10.123.123.140:57680 Fingerprint: SHA256:rcLnaYptHJVEGdltjzt1P0isyRNB9pi02rpJKCc41RA is not a certificate
2021/09/24 15:42:31 modules/ssh/ssh.go:248:publicKeyHandler() [D] Successfully authenticated: 10.123.123.140:57680 Public Key Fingerprint: SHA256:rcLnaYptHJVEGdltjzt1P0isyRNB9pi02rpJKCc41RA

[wxiaoguang]

I just tried, can not reproduce this problem on my side (Gitea 1.15.3 + Windows)

May you get a full stack trace? It would help much.

Steps:

1. Set app.ini:

[server]
ENABLE_PPROF = true

2. Restart Gitea

3. Try to trigger the bug, when the processes get stuck for a while, use curl or browser to visit: http://127.0.0.1:6060/debug/pprof/goroutine?debug=1

4. Report the output stack trace here (the stack trace doesn't contain sensitive data).

[zeripath]

That process list indicates to me that there's a deadlock somewhere in the hook.

I'm confused as to how or where.

I guess we're going to need to add some extra logging into:

gitea/routers/private/hook.go

Lines 133 to 399 in 7ce938b

	gitRepo, err := git.OpenRepository(repo.RepoPath())
	if err != nil {
	log.Error("Unable to get git repository for: %s/%s Error: %v", ownerName, repoName, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: err.Error(),
	})
	return
	}
	defer gitRepo.Close()
	// Generate git environment for checking commits
	env := os.Environ()
	if opts.GitAlternativeObjectDirectories != "" {
	env = append(env,
	private.GitAlternativeObjectDirectories+"="+opts.GitAlternativeObjectDirectories)
	}
	if opts.GitObjectDirectory != "" {
	env = append(env,
	private.GitObjectDirectory+"="+opts.GitObjectDirectory)
	}
	if opts.GitQuarantinePath != "" {
	env = append(env,
	private.GitQuarantinePath+"="+opts.GitQuarantinePath)
	}
	protectedTags, err := repo.GetProtectedTags()
	if err != nil {
	log.Error("Unable to get protected tags for %-v Error: %v", repo, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: err.Error(),
	})
	return
	}
	// Iterate across the provided old commit IDs
	for i := range opts.OldCommitIDs {
	oldCommitID := opts.OldCommitIDs[i]
	newCommitID := opts.NewCommitIDs[i]
	refFullName := opts.RefFullNames[i]
	if strings.HasPrefix(refFullName, git.BranchPrefix) {
	branchName := strings.TrimPrefix(refFullName, git.BranchPrefix)
	if branchName == repo.DefaultBranch && newCommitID == git.EmptySHA {
	log.Warn("Forbidden: Branch: %s is the default branch in %-v and cannot be deleted", branchName, repo)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("branch %s is the default branch and cannot be deleted", branchName),
	})
	return
	}
	protectBranch, err := models.GetProtectedBranchBy(repo.ID, branchName)
	if err != nil {
	log.Error("Unable to get protected branch: %s in %-v Error: %v", branchName, repo, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: err.Error(),
	})
	return
	}
	// Allow pushes to non-protected branches
	if protectBranch == nil || !protectBranch.IsProtected() {
	continue
	}
	// This ref is a protected branch.
	//
	// First of all we need to enforce absolutely:
	//
	// 1. Detect and prevent deletion of the branch
	if newCommitID == git.EmptySHA {
	log.Warn("Forbidden: Branch: %s in %-v is protected from deletion", branchName, repo)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("branch %s is protected from deletion", branchName),
	})
	return
	}
	// 2. Disallow force pushes to protected branches
	if git.EmptySHA != oldCommitID {
	output, err := git.NewCommand("rev-list", "--max-count=1", oldCommitID, "^"+newCommitID).RunInDirWithEnv(repo.RepoPath(), env)
	if err != nil {
	log.Error("Unable to detect force push between: %s and %s in %-v Error: %v", oldCommitID, newCommitID, repo, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Fail to detect force push: %v", err),
	})
	return
	} else if len(output) > 0 {
	log.Warn("Forbidden: Branch: %s in %-v is protected from force push", branchName, repo)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("branch %s is protected from force push", branchName),
	})
	return
	}
	}
	// 3. Enforce require signed commits
	if protectBranch.RequireSignedCommits {
	err := verifyCommits(oldCommitID, newCommitID, gitRepo, env)
	if err != nil {
	if !isErrUnverifiedCommit(err) {
	log.Error("Unable to check commits from %s to %s in %-v: %v", oldCommitID, newCommitID, repo, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Unable to check commits from %s to %s: %v", oldCommitID, newCommitID, err),
	})
	return
	}
	unverifiedCommit := err.(*errUnverifiedCommit).sha
	log.Warn("Forbidden: Branch: %s in %-v is protected from unverified commit %s", branchName, repo, unverifiedCommit)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("branch %s is protected from unverified commit %s", branchName, unverifiedCommit),
	})
	return
	}
	}
	// Now there are several tests which can be overridden:
	//
	// 4. Check protected file patterns - this is overridable from the UI
	changedProtectedfiles := false
	protectedFilePath := ""
	globs := protectBranch.GetProtectedFilePatterns()
	if len(globs) > 0 {
	_, err := pull_service.CheckFileProtection(oldCommitID, newCommitID, globs, 1, env, gitRepo)
	if err != nil {
	if !models.IsErrFilePathProtected(err) {
	log.Error("Unable to check file protection for commits from %s to %s in %-v: %v", oldCommitID, newCommitID, repo, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Unable to check file protection for commits from %s to %s: %v", oldCommitID, newCommitID, err),
	})
	return
	}
	changedProtectedfiles = true
	protectedFilePath = err.(models.ErrFilePathProtected).Path
	}
	}
	// 5. Check if the doer is allowed to push
	canPush := false
	if opts.IsDeployKey {
	canPush = !changedProtectedfiles && protectBranch.CanPush && (!protectBranch.EnableWhitelist || protectBranch.WhitelistDeployKeys)
	} else {
	canPush = !changedProtectedfiles && protectBranch.CanUserPush(opts.UserID)
	}
	// 6. If we're not allowed to push directly
	if !canPush {
	// Is this is a merge from the UI/API?
	if opts.PullRequestID == 0 {
	// 6a. If we're not merging from the UI/API then there are two ways we got here:
	//
	// We are changing a protected file and we're not allowed to do that
	if changedProtectedfiles {
	log.Warn("Forbidden: Branch: %s in %-v is protected from changing file %s", branchName, repo, protectedFilePath)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("branch %s is protected from changing file %s", branchName, protectedFilePath),
	})
	return
	}
	// Or we're simply not able to push to this protected branch
	log.Warn("Forbidden: User %d is not allowed to push to protected branch: %s in %-v", opts.UserID, branchName, repo)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("Not allowed to push to protected branch %s", branchName),
	})
	return
	}
	// 6b. Merge (from UI or API)
	// Get the PR, user and permissions for the user in the repository
	pr, err := models.GetPullRequestByID(opts.PullRequestID)
	if err != nil {
	log.Error("Unable to get PullRequest %d Error: %v", opts.PullRequestID, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Unable to get PullRequest %d Error: %v", opts.PullRequestID, err),
	})
	return
	}
	user, err := models.GetUserByID(opts.UserID)
	if err != nil {
	log.Error("Unable to get User id %d Error: %v", opts.UserID, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Unable to get User id %d Error: %v", opts.UserID, err),
	})
	return
	}
	perm, err := models.GetUserRepoPermission(repo, user)
	if err != nil {
	log.Error("Unable to get Repo permission of repo %s/%s of User %s", repo.OwnerName, repo.Name, user.Name, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Unable to get Repo permission of repo %s/%s of User %s: %v", repo.OwnerName, repo.Name, user.Name, err),
	})
	return
	}
	// Now check if the user is allowed to merge PRs for this repository
	allowedMerge, err := pull_service.IsUserAllowedToMerge(pr, perm, user)
	if err != nil {
	log.Error("Error calculating if allowed to merge: %v", err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Error calculating if allowed to merge: %v", err),
	})
	return
	}
	if !allowedMerge {
	log.Warn("Forbidden: User %d is not allowed to push to protected branch: %s in %-v and is not allowed to merge pr #%d", opts.UserID, branchName, repo, pr.Index)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("Not allowed to push to protected branch %s", branchName),
	})
	return
	}
	// If we're an admin for the repository we can ignore status checks, reviews and override protected files
	if perm.IsAdmin() {
	continue
	}
	// Now if we're not an admin - we can't overwrite protected files so fail now
	if changedProtectedfiles {
	log.Warn("Forbidden: Branch: %s in %-v is protected from changing file %s", branchName, repo, protectedFilePath)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("branch %s is protected from changing file %s", branchName, protectedFilePath),
	})
	return
	}
	// Check all status checks and reviews are ok
	if err := pull_service.CheckPRReadyToMerge(pr, true); err != nil {
	if models.IsErrNotAllowedToMerge(err) {
	log.Warn("Forbidden: User %d is not allowed push to protected branch %s in %-v and pr #%d is not ready to be merged: %s", opts.UserID, branchName, repo, pr.Index, err.Error())
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("Not allowed to push to protected branch %s and pr #%d is not ready to be merged: %s", branchName, opts.PullRequestID, err.Error()),
	})
	return
	}
	log.Error("Unable to check if mergable: protected branch %s in %-v and pr #%d. Error: %v", opts.UserID, branchName, repo, pr.Index, err)
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: fmt.Sprintf("Unable to get status of pull request %d. Error: %v", opts.PullRequestID, err),
	})
	return
	}
	}
	} else if strings.HasPrefix(refFullName, git.TagPrefix) {
	tagName := strings.TrimPrefix(refFullName, git.TagPrefix)
	isAllowed, err := models.IsUserAllowedToControlTag(protectedTags, tagName, opts.UserID)
	if err != nil {
	ctx.JSON(http.StatusInternalServerError, private.Response{
	Err: err.Error(),
	})
	return
	}
	if !isAllowed {
	log.Warn("Forbidden: Tag %s in %-v is protected", tagName, repo)
	ctx.JSON(http.StatusForbidden, private.Response{
	Err: fmt.Sprintf("Tag %s is protected", tagName),
	})
	return
	}
	}
	}
	ctx.PlainText(http.StatusOK, []byte("ok"))
	}

[zeripath]

@dannyzcy if I were able to provide patches would you be able to recompile gitea and report the logs? The PPROF results that @wxiaoguang mentioned would also be helpful.

[dannyzcy]

@wxiaoguang The logs is as follows:

http://127.0.0.1:6060/debug/pprof/goroutine?debug=1

goroutine profile: total 45
4 @ 0x10e041a 0x10f1468 0x26f7cb9 0x1116541
#	0x26f7cb8	github.com/blevesearch/bleve_index_api.AnalysisWorker+0x98	/source/vendor/github.com/blevesearch/bleve_index_api/analysis.go:46

3 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119c176 0x11d8fd6 0x11ebed8 0x146d7ff 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64		/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119c175	internal/poll.(*FD).Read+0x2f5			/usr/local/go/src/internal/poll/fd_windows.go:441
#	0x11d8fd5	net.(*netFD).Read+0x55				/usr/local/go/src/net/fd_posix.go:55
#	0x11ebed7	net.(*conn).Read+0x97				/usr/local/go/src/net/net.go:183
#	0x146d7fe	net/http.(*connReader).backgroundRead+0x5e	/usr/local/go/src/net/http/server.go:672

3 @ 0x10e041a 0x10f1468 0x2f6b4d1 0x2d409dc 0x14756cb 0x23d432b 0x14756cb 0x2d410ca 0x14756cb 0x2d410ca 0x14756cb 0x2d410ca 0x14756cb 0x2d64782 0x14756cb 0x2d410ca 0x14756cb 0x23f55ca 0x14756cb 0x23d1e2b 0x23d82dc 0x14756cb 0x23d432b 0x14756cb 0x2f76c58 0x14756cb 0x21d5f2f 0x14756cb 0x23474f1 0x14756cb 0x23d1e2b 0x23d82dc
#	0x2f6b4d0	code.gitea.io/gitea/routers/web/events.Events+0x7b0		/source/routers/web/events/events.go:77
#	0x2d409db	code.gitea.io/gitea/modules/web.Wrap.func1+0x1fb		/source/modules/web/route.go:64
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a			/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d64781	github.com/go-chi/chi/middleware.GetHead.func1+0x181		/source/vendor/github.com/go-chi/chi/middleware/get_head.go:37
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2d410c9	code.gitea.io/gitea/modules/web.Middle.func1.1+0x149		/source/modules/web/route.go:103
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23f55c9	code.gitea.io/gitea/modules/context.Contexter.func1.1+0x1a69	/source/modules/context/context.go:776
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa			/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b			/source/vendor/github.com/go-chi/chi/mux.go:311
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a			/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x2f76c57	code.gitea.io/gitea/routers/web.Recovery.func1.1+0x97		/source/routers/web/base.go:186
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x21d5f2e	gitea.com/go-chi/session.Sessioner.func1.1+0x24e		/source/vendor/gitea.com/go-chi/session/session.go:256
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23474f0	code.gitea.io/gitea/modules/public.AssetsHandler.func1.1+0x90	/source/modules/public/public.go:42
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a				/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa			/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b			/source/vendor/github.com/go-chi/chi/mux.go:311

2 @ 0x10a4d55 0x1112e12 0x116e1c5 0x1165eaf 0x119c0a5 0x11a5b05 0x11a5ad1 0x115c0cc 0x12f432a 0x12f42fa 0x12f43ce 0x1116541
#	0x1112e11	syscall.Syscall6+0xf1				/usr/local/go/src/runtime/syscall_windows.go:343
#	0x116e1c4	syscall.ReadFile+0x104				/usr/local/go/src/syscall/zsyscall_windows.go:1006
#	0x1165eae	syscall.Read+0x6e				/usr/local/go/src/syscall/syscall_windows.go:369
#	0x119c0a4	internal/poll.(*FD).Read+0x224			/usr/local/go/src/internal/poll/fd_windows.go:427
#	0x11a5b04	os.(*File).read+0x84				/usr/local/go/src/os/file_posix.go:31
#	0x11a5ad0	os.(*File).Read+0x50				/usr/local/go/src/os/file.go:117
#	0x115c0cb	io.copyBuffer+0x12b				/usr/local/go/src/io/io.go:423
#	0x12f4329	io.Copy+0x69					/usr/local/go/src/io/io.go:382
#	0x12f42f9	os/exec.(*Cmd).writerDescriptor.func1+0x39	/usr/local/go/src/os/exec/exec.go:311
#	0x12f43cd	os/exec.(*Cmd).Start.func1+0x2d			/usr/local/go/src/os/exec/exec.go:441

2 @ 0x10e041a 0x10f1468 0x14b7dec 0x1116541
#	0x14b7deb	code.gitea.io/gitea/modules/log.(*ChannelledLog).Start+0xab	/source/modules/log/event.go:70

2 @ 0x10e041a 0x10f1468 0x14b93a5 0x1116541
#	0x14b93a4	code.gitea.io/gitea/modules/log.(*MultiChannelledLog).Start+0x364	/source/modules/log/event.go:300

2 @ 0x10e041a 0x10f1468 0x192a74a 0x1116541
#	0x192a749	code.gitea.io/gitea/modules/graceful.(*Server).awaitShutdown+0xe9	/source/modules/graceful/server_hooks.go:16

1 @ 0x10ae5d3 0x11118ef 0x19252c9 0x1116541
#	0x11118ee	os/signal.signal_recv+0xae	/usr/local/go/src/runtime/sigqueue.go:168
#	0x19252c8	os/signal.loop+0x28		/usr/local/go/src/os/signal/signal_unix.go:23

1 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x12f21cc 0x16be80a 0x16bf893 0x16bf7f7 0x16bf7f8 0x1700f0f 0x1116541
#	0x12f21cb	os/exec.(*Cmd).Wait+0x14b								/usr/local/go/src/os/exec/exec.go:515
#	0x16be809	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipelineFunc+0x4e9	/source/modules/git/command.go:168
#	0x16bf892	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipeline+0xd2		/source/modules/git/command.go:107
#	0x16bf7f6	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutFullPipeline+0x36		/source/modules/git/command.go:184
#	0x16bf7f7	code.gitea.io/gitea/modules/git.(*Command).RunInDirFullPipeline+0x37			/source/modules/git/command.go:220
#	0x1700f0e	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func2+0x3ae				/source/modules/git/batch_reader.go:52

1 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x15b2b51 0x15b1f14 0x1116541
#	0x15b2b50	golang.org/x/crypto/ssh.(*handshakeTransport).readOnePacket+0x1d0	/source/vendor/golang.org/x/crypto/ssh/handshake.go:410
#	0x15b1f13	golang.org/x/crypto/ssh.(*handshakeTransport).readLoop+0x53		/source/vendor/golang.org/x/crypto/ssh/handshake.go:197

1 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x1700b46 0x16dc6f4 0x2eb31d4 0x2d40b9c 0x14756cb 0x23d432b 0x14756cb 0x2ec0aab 0x14756cb 0x23f6f30 0x14756cb 0x23d1e2b 0x23d82dc 0x14756cb 0x23d432b 0x14756cb 0x2d69305 0x14756cb 0x2d68529 0x14756cb 0x2d650c8 0x14756cb 0x2d605f4 0x14756cb 0x2d68e48 0x14756cb 0x23d1bb1 0x2d3fefb 0x1ede55b
#	0x1700b45	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func1+0xe5		/source/modules/git/batch_reader.go:42
#	0x16dc6f3	code.gitea.io/gitea/modules/git.(*Repository).Close+0x53		/source/modules/git/repo_base_nogogit.go:86
#	0x2eb31d3	code.gitea.io/gitea/routers/private.HookPreReceive+0x4553		/source/routers/private/hook.go:399
#	0x2d40b9b	code.gitea.io/gitea/modules/web.Wrap.func1+0x3bb			/source/modules/web/route.go:76
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2ec0aaa	code.gitea.io/gitea/routers/private.CheckInternalToken.func1+0x22a	/source/routers/private/internal.go:31
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23f6f2f	code.gitea.io/gitea/modules/context.PrivateContexter.func1.1+0x16f	/source/modules/context/private.go:42
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa				/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b				/source/vendor/github.com/go-chi/chi/mux.go:311
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d69304	code.gitea.io/gitea/routers/common.Middlewares.func2.1+0x84		/source/routers/common/middleware.go:72
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68528	code.gitea.io/gitea/routers/common.LoggerHandler.func1.1+0x368		/source/routers/common/logger.go:23
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d650c7	github.com/go-chi/chi/middleware.StripSlashes.func1+0x107		/source/vendor/github.com/go-chi/chi/middleware/strip.go:30
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d605f3	github.com/chi-middleware/proxy.ForwardedHeaders.func1.1+0xf3		/source/vendor/github.com/chi-middleware/proxy/middleware.go:37
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68e47	code.gitea.io/gitea/routers/common.Middlewares.func1.1+0x67		/source/routers/common/middleware.go:25
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d1bb0	github.com/go-chi/chi.(*Mux).ServeHTTP+0x330				/source/vendor/github.com/go-chi/chi/mux.go:87
#	0x2d3fefa	code.gitea.io/gitea/modules/web.(*Route).ServeHTTP+0x5a			/source/modules/web/route.go:318
#	0x1ede55a	github.com/gorilla/context.ClearHandler.func1+0x7a			/source/vendor/github.com/gorilla/context/context.go:141

1 @ 0x10e041a 0x10a7f3a 0x10a7bab 0x15b1d15 0x15b1ce9 0x15cbed0 0x15cb86e 0x2d2fcc9 0x1116541
#	0x15b1d14	golang.org/x/crypto/ssh.(*handshakeTransport).readPacket+0x54	/source/vendor/golang.org/x/crypto/ssh/handshake.go:187
#	0x15b1ce8	golang.org/x/crypto/ssh.(*handshakeTransport).waitSession+0x28	/source/vendor/golang.org/x/crypto/ssh/handshake.go:154
#	0x15cbecf	golang.org/x/crypto/ssh.(*connection).serverHandshake+0x28f	/source/vendor/golang.org/x/crypto/ssh/server.go:251
#	0x15cb86d	golang.org/x/crypto/ssh.NewServerConn+0x18d			/source/vendor/golang.org/x/crypto/ssh/server.go:206
#	0x2d2fcc8	github.com/gliderlabs/ssh.(*Server).HandleConn+0x188		/source/vendor/github.com/gliderlabs/ssh/server.go:281

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119c176 0x11d8fd6 0x11ebed8 0x146dd7f 0x12daf6d 0x12dbd65 0x12dbfbb 0x13faa0b 0x1467f31 0x1467f32 0x146f26b 0x1473fc5 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64		/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119c175	internal/poll.(*FD).Read+0x2f5			/usr/local/go/src/internal/poll/fd_windows.go:441
#	0x11d8fd5	net.(*netFD).Read+0x55				/usr/local/go/src/net/fd_posix.go:55
#	0x11ebed7	net.(*conn).Read+0x97				/usr/local/go/src/net/net.go:183
#	0x146dd7e	net/http.(*connReader).Read+0x1be		/usr/local/go/src/net/http/server.go:780
#	0x12daf6c	bufio.(*Reader).fill+0x10c			/usr/local/go/src/bufio/bufio.go:101
#	0x12dbd64	bufio.(*Reader).ReadSlice+0x44			/usr/local/go/src/bufio/bufio.go:360
#	0x12dbfba	bufio.(*Reader).ReadLine+0x3a			/usr/local/go/src/bufio/bufio.go:389
#	0x13faa0a	net/textproto.(*Reader).readLineSlice+0xea	/usr/local/go/src/net/textproto/reader.go:57
#	0x1467f30	net/textproto.(*Reader).ReadLine+0xb0		/usr/local/go/src/net/textproto/reader.go:38
#	0x1467f31	net/http.readRequest+0xb1			/usr/local/go/src/net/http/request.go:1027
#	0x146f26a	net/http.(*conn).readRequest+0x1aa		/usr/local/go/src/net/http/server.go:966
#	0x1473fc4	net/http.(*conn).serve+0x704			/usr/local/go/src/net/http/server.go:1858

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119c176 0x11d8fd6 0x11ebed8 0x2d2d7ca 0x12db658 0x115bb4e 0x15abb47 0x15abaf7 0x15d272f 0x15d2645 0x15b867b 0x15b458d 0x15b3e0f 0x15b244f 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64					/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b					/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112						/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119c175	internal/poll.(*FD).Read+0x2f5						/usr/local/go/src/internal/poll/fd_windows.go:441
#	0x11d8fd5	net.(*netFD).Read+0x55							/usr/local/go/src/net/fd_posix.go:55
#	0x11ebed7	net.(*conn).Read+0x97							/usr/local/go/src/net/net.go:183
#	0x2d2d7c9	github.com/gliderlabs/ssh.(*serverConn).Read+0x69			/source/vendor/github.com/gliderlabs/ssh/conn.go:28
#	0x12db657	bufio.(*Reader).Read+0x237						/usr/local/go/src/bufio/bufio.go:227
#	0x115bb4d	io.ReadAtLeast+0x8d							/usr/local/go/src/io/io.go:328
#	0x15abb46	io.ReadFull+0x86							/usr/local/go/src/io/io.go:347
#	0x15abaf6	golang.org/x/crypto/ssh.(*streamPacketCipher).readCipherPacket+0x36	/source/vendor/golang.org/x/crypto/ssh/cipher.go:153
#	0x15d272e	golang.org/x/crypto/ssh.(*connectionState).readPacket+0x6e		/source/vendor/golang.org/x/crypto/ssh/transport.go:130
#	0x15d2644	golang.org/x/crypto/ssh.(*transport).readPacket+0x44			/source/vendor/golang.org/x/crypto/ssh/transport.go:114
#	0x15b867a	golang.org/x/crypto/ssh.(*curve25519sha256).Server+0x5a			/source/vendor/golang.org/x/crypto/ssh/kex.go:490
#	0x15b458c	golang.org/x/crypto/ssh.(*handshakeTransport).server+0x1ac		/source/vendor/golang.org/x/crypto/ssh/handshake.go:622
#	0x15b3e0e	golang.org/x/crypto/ssh.(*handshakeTransport).enterKeyExchange+0x38e	/source/vendor/golang.org/x/crypto/ssh/handshake.go:585
#	0x15b244e	golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop+0x1ae		/source/vendor/golang.org/x/crypto/ssh/handshake.go:301

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119e69c 0x119e968 0x11da9c5 0x11f3c39 0x11f282c 0x1929fc7 0x14790fc 0x1929adc 0x1928b7a 0x192af65 0x30a896f 0x30a88f9 0x30a79b9 0x241f0d0 0x241ceb5 0x30e4dca 0x10dfff6 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64					/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b					/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112						/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119e69b	internal/poll.(*FD).acceptOne+0x9b					/usr/local/go/src/internal/poll/fd_windows.go:810
#	0x119e967	internal/poll.(*FD).Accept+0x167					/usr/local/go/src/internal/poll/fd_windows.go:844
#	0x11da9c4	net.(*netFD).accept+0x84						/usr/local/go/src/net/fd_windows.go:139
#	0x11f3c38	net.(*TCPListener).accept+0x38						/usr/local/go/src/net/tcpsock_posix.go:139
#	0x11f282b	net.(*TCPListener).AcceptTCP+0x6b					/usr/local/go/src/net/tcpsock.go:248
#	0x1929fc6	code.gitea.io/gitea/modules/graceful.(*wrappedListener).Accept+0x66	/source/modules/graceful/server.go:215
#	0x14790fb	net/http.(*Server).Serve+0x29b						/usr/local/go/src/net/http/server.go:2961
#	0x1929adb	code.gitea.io/gitea/modules/graceful.(*Server).Serve+0x17b		/source/modules/graceful/server.go:168
#	0x1928b79	code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe+0x1f9	/source/modules/graceful/server.go:96
#	0x192af64	code.gitea.io/gitea/modules/graceful.HTTPListenAndServe+0xa4		/source/modules/graceful/server_http.go:33
#	0x30a896e	code.gitea.io/gitea/cmd.runHTTP+0x58e					/source/cmd/web_graceful.go:20
#	0x30a88f8	code.gitea.io/gitea/cmd.listen+0x518					/source/cmd/web.go:208
#	0x30a79b8	code.gitea.io/gitea/cmd.runWeb+0x4f8					/source/cmd/web.go:159
#	0x241f0cf	github.com/urfave/cli.HandleAction+0x10f				/source/vendor/github.com/urfave/cli/app.go:524
#	0x241ceb4	github.com/urfave/cli.(*App).Run+0x674					/source/vendor/github.com/urfave/cli/app.go:286
#	0x30e4dc9	main.main+0xa69								/source/main.go:115
#	0x10dfff5	runtime.main+0x255							/usr/local/go/src/runtime/proc.go:225

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119e69c 0x119e968 0x11da9c5 0x11f3c39 0x11f282c 0x1929fc7 0x2d2f893 0x1929adc 0x1928b7a 0x2d39038 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64					/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b					/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112						/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119e69b	internal/poll.(*FD).acceptOne+0x9b					/usr/local/go/src/internal/poll/fd_windows.go:810
#	0x119e967	internal/poll.(*FD).Accept+0x167					/usr/local/go/src/internal/poll/fd_windows.go:844
#	0x11da9c4	net.(*netFD).accept+0x84						/usr/local/go/src/net/fd_windows.go:139
#	0x11f3c38	net.(*TCPListener).accept+0x38						/usr/local/go/src/net/tcpsock_posix.go:139
#	0x11f282b	net.(*TCPListener).AcceptTCP+0x6b					/usr/local/go/src/net/tcpsock.go:248
#	0x1929fc6	code.gitea.io/gitea/modules/graceful.(*wrappedListener).Accept+0x66	/source/modules/graceful/server.go:215
#	0x2d2f892	github.com/gliderlabs/ssh.(*Server).Serve+0x152				/source/vendor/github.com/gliderlabs/ssh/server.go:237
#	0x1929adb	code.gitea.io/gitea/modules/graceful.(*Server).Serve+0x17b		/source/modules/graceful/server.go:168
#	0x1928b79	code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe+0x1f9	/source/modules/graceful/server.go:96
#	0x2d39037	code.gitea.io/gitea/modules/ssh.listen+0xb7				/source/modules/ssh/ssh_graceful.go:20

1 @ 0x10e041a 0x10d5fc5 0x110f785 0x11999cc 0x119af53 0x119e69c 0x119e968 0x11da9c5 0x11f3c39 0x11f29eb 0x14790fc 0x1478e05 0x30aaf1c 0x30aaf1d 0x1116541
#	0x110f784	internal/poll.runtime_pollWait+0x64		/usr/local/go/src/runtime/netpoll.go:222
#	0x11999cb	internal/poll.(*pollDesc).wait+0x4b		/usr/local/go/src/internal/poll/fd_poll_runtime.go:87
#	0x119af52	internal/poll.execIO+0x112			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0x119e69b	internal/poll.(*FD).acceptOne+0x9b		/usr/local/go/src/internal/poll/fd_windows.go:810
#	0x119e967	internal/poll.(*FD).Accept+0x167		/usr/local/go/src/internal/poll/fd_windows.go:844
#	0x11da9c4	net.(*netFD).accept+0x84			/usr/local/go/src/net/fd_windows.go:139
#	0x11f3c38	net.(*TCPListener).accept+0x38			/usr/local/go/src/net/tcpsock_posix.go:139
#	0x11f29ea	net.(*TCPListener).Accept+0x6a			/usr/local/go/src/net/tcpsock.go:261
#	0x14790fb	net/http.(*Server).Serve+0x29b			/usr/local/go/src/net/http/server.go:2961
#	0x1478e04	net/http.(*Server).ListenAndServe+0xc4		/usr/local/go/src/net/http/server.go:2890
#	0x30aaf1b	net/http.ListenAndServe+0x9b			/usr/local/go/src/net/http/server.go:3144
#	0x30aaf1c	code.gitea.io/gitea/cmd.runWeb.func2+0x9c	/source/cmd/web.go:142

1 @ 0x10e041a 0x10f1468 0x150381b 0x1116541
#	0x150381a	database/sql.(*DB).connectionCleaner+0x11a	/usr/local/go/src/database/sql/sql.go:1005

1 @ 0x10e041a 0x10f1468 0x15042bc 0x1116541
#	0x15042bb	database/sql.(*DB).connectionOpener+0xbb	/usr/local/go/src/database/sql/sql.go:1133

1 @ 0x10e041a 0x10f1468 0x18243ca 0x1116541
#	0x18243c9	github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain+0x109	/source/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206

1 @ 0x10e041a 0x10f1468 0x184afe5 0x1116541
#	0x184afe4	github.com/syndtr/goleveldb/leveldb.(*DB).compactionError+0xc4	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90

1 @ 0x10e041a 0x10f1468 0x18505cb 0x1116541
#	0x18505ca	github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction+0x12a	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:762

1 @ 0x10e041a 0x10f1468 0x18508cf 0x1116541
#	0x18508ce	github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction+0x22e	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:825

1 @ 0x10e041a 0x10f1468 0x1854a7b 0x1116541
#	0x1854a7a	github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain+0xda	/source/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:101

1 @ 0x10e041a 0x10f1468 0x192595b 0x1116541
#	0x192595a	golang.org/x/sys/windows/svc/debug.Run.func1+0xba	/source/vendor/golang.org/x/sys/windows/svc/debug/service.go:32

1 @ 0x10e041a 0x10f1468 0x19277bf 0x1925837 0x192c2fb 0x1116541
#	0x19277be	code.gitea.io/gitea/modules/graceful.(*Manager).Execute+0x23e		/source/modules/graceful/manager_windows.go:132
#	0x1925836	golang.org/x/sys/windows/svc/debug.Run+0x156				/source/vendor/golang.org/x/sys/windows/svc/debug/service.go:40
#	0x192c2fa	code.gitea.io/gitea/modules/graceful.(*Manager).start.func1+0x5a	/source/modules/graceful/manager_windows.go:99

1 @ 0x10e041a 0x10f1468 0x1f36627 0x1116541
#	0x1f36626	github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1+0xa6	/source/vendor/github.com/go-sql-driver/mysql/connection.go:614

1 @ 0x10e041a 0x10f1468 0x25d040a 0x1925ed0 0x1116541
#	0x25d0409	code.gitea.io/gitea/services/webhook.DeliverHooks+0x269				/source/services/webhook/deliver.go:221
#	0x1925ecf	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0x10e041a 0x10f1468 0x2b502e8 0x2b4fba5 0x1925c38 0x1116541
#	0x2b502e7	code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).readToChan+0x427		/source/modules/queue/queue_bytefifo.go:152
#	0x2b4fba4	code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).Run+0x2a4			/source/modules/queue/queue_bytefifo.go:121
#	0x1925c37	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownFns+0x117	/source/modules/graceful/manager.go:83

1 @ 0x10e041a 0x10f1468 0x2d13a7e 0x1116541
#	0x2d13a7d	github.com/gogs/cron.(*Cron).run+0x29d	/source/vendor/github.com/gogs/cron/cron.go:197

1 @ 0x10e041a 0x10f1468 0x2d18950 0x1925ed0 0x1116541
#	0x2d1894f	code.gitea.io/gitea/services/mirror.syncMirrors+0xef				/source/services/mirror/mirror.go:72
#	0x1925ecf	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0x10e041a 0x10f1468 0x2d29f1a 0x1925ed0 0x1116541
#	0x2d29f19	code.gitea.io/gitea/modules/eventsource.(*Manager).Run+0xf9			/source/modules/eventsource/manager_run.go:32
#	0x1925ecf	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0x110f345 0x2cd88d5 0x2cd8687 0x2cd5158 0x2d10925 0x2d121a9 0x14756cb 0x1477597 0x1478d0a 0x147418d 0x1116541
#	0x110f344	runtime/pprof.runtime_goroutineProfileWithLabels+0x64	/usr/local/go/src/runtime/mprof.go:716
#	0x2cd88d4	runtime/pprof.writeRuntimeProfile+0xd4			/usr/local/go/src/runtime/pprof/pprof.go:724
#	0x2cd8686	runtime/pprof.writeGoroutine+0xa6			/usr/local/go/src/runtime/pprof/pprof.go:684
#	0x2cd5157	runtime/pprof.(*Profile).WriteTo+0x3f7			/usr/local/go/src/runtime/pprof/pprof.go:331
#	0x2d10924	net/http/pprof.handler.ServeHTTP+0x384			/usr/local/go/src/net/http/pprof/pprof.go:253
#	0x2d121a8	net/http/pprof.Index+0x8e8				/usr/local/go/src/net/http/pprof/pprof.go:371
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a			/usr/local/go/src/net/http/server.go:2049
#	0x1477596	net/http.(*ServeMux).ServeHTTP+0x1b6			/usr/local/go/src/net/http/server.go:2428
#	0x1478d09	net/http.serverHandler.ServeHTTP+0xa9			/usr/local/go/src/net/http/server.go:2867
#	0x147418c	net/http.(*conn).serve+0x8cc				/usr/local/go/src/net/http/server.go:1932

1 @ 0x146d7a1 0x1116541
#	0x146d7a0	net/http.(*connReader).backgroundRead+0x0	/usr/local/go/src/net/http/server.go:671

[dannyzcy]

@dannyzcy if I were able to provide patches would you be able to recompile gitea and report the logs?

I can't always reproduce this bug

sometime appear sometime disappear

and not always the same repository

I don't know how to recompile gitea

sorry

[lunny]

Is it blocked by internal push update router?

[dannyzcy]

New discovery different source: services/pull/pull.go:212

PPROF results:

goroutine profile: total 55
8 @ 0x10a4d55 0x1112e12 0x116e1c5 0x1165eaf 0x119c0a5 0x11a5b05 0x11a5ad1 0x115c0cc 0x12f432a 0x12f42fa 0x12f43ce 0x1116541
#	0x1112e11	syscall.Syscall6+0xf1				/usr/local/go/src/runtime/syscall_windows.go:343
#	0x116e1c4	syscall.ReadFile+0x104				/usr/local/go/src/syscall/zsyscall_windows.go:1006
#	0x1165eae	syscall.Read+0x6e				/usr/local/go/src/syscall/syscall_windows.go:369
#	0x119c0a4	internal/poll.(*FD).Read+0x224			/usr/local/go/src/internal/poll/fd_windows.go:427
#	0x11a5b04	os.(*File).read+0x84				/usr/local/go/src/os/file_posix.go:31
#	0x11a5ad0	os.(*File).Read+0x50				/usr/local/go/src/os/file.go:117
#	0x115c0cb	io.copyBuffer+0x12b				/usr/local/go/src/io/io.go:423
#	0x12f4329	io.Copy+0x69					/usr/local/go/src/io/io.go:382
#	0x12f42f9	os/exec.(*Cmd).writerDescriptor.func1+0x39	/usr/local/go/src/os/exec/exec.go:311
#	0x12f43cd	os/exec.(*Cmd).Start.func1+0x2d			/usr/local/go/src/os/exec/exec.go:441

4 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x12f21cc 0x16be80a 0x16bf893 0x16bf7f7 0x16bf7f8 0x1700f0f 0x1116541
#	0x12f21cb	os/exec.(*Cmd).Wait+0x14b								/usr/local/go/src/os/exec/exec.go:515
#	0x16be809	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipelineFunc+0x4e9	/source/modules/git/command.go:168
#	0x16bf892	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipeline+0xd2		/source/modules/git/command.go:107
#	0x16bf7f6	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutFullPipeline+0x36		/source/modules/git/command.go:184
#	0x16bf7f7	code.gitea.io/gitea/modules/git.(*Command).RunInDirFullPipeline+0x37			/source/modules/git/command.go:220
#	0x1700f0e	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func2+0x3ae				/source/modules/git/batch_reader.go:52

4 @ 0x10e041a 0x10f1468 0x26f7cb9 0x1116541
#	0x26f7cb8	github.com/blevesearch/bleve_index_api.AnalysisWorker+0x98	/source/vendor/github.com/blevesearch/bleve_index_api/analysis.go:46

3 @ 0x10e041a 0x10a7f3a 0x10a7b6b 0x1700b46 0x16dc6f4 0x2eb31d4 0x2d40b9c 0x14756cb 0x23d432b 0x14756cb 0x2ec0aab 0x14756cb 0x23f6f30 0x14756cb 0x23d1e2b 0x23d82dc 0x14756cb 0x23d432b 0x14756cb 0x2d69305 0x14756cb 0x2d68529 0x14756cb 0x2d650c8 0x14756cb 0x2d605f4 0x14756cb 0x2d68e48 0x14756cb 0x23d1bb1 0x2d3fefb 0x1ede55b
#	0x1700b45	code.gitea.io/gitea/modules/git.CatFileBatchCheck.func1+0xe5		/source/modules/git/batch_reader.go:42
#	0x16dc6f3	code.gitea.io/gitea/modules/git.(*Repository).Close+0x53		/source/modules/git/repo_base_nogogit.go:86
#	0x2eb31d3	code.gitea.io/gitea/routers/private.HookPreReceive+0x4553		/source/routers/private/hook.go:399
#	0x2d40b9b	code.gitea.io/gitea/modules/web.Wrap.func1+0x3bb			/source/modules/web/route.go:76
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2ec0aaa	code.gitea.io/gitea/routers/private.CheckInternalToken.func1+0x22a	/source/routers/private/internal.go:31
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23f6f2f	code.gitea.io/gitea/modules/context.PrivateContexter.func1.1+0x16f	/source/modules/context/private.go:42
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d1e2a	github.com/go-chi/chi.(*Mux).ServeHTTP+0x5aa				/source/vendor/github.com/go-chi/chi/mux.go:70
#	0x23d82db	github.com/go-chi/chi.(*Mux).Mount.func1+0x17b				/source/vendor/github.com/go-chi/chi/mux.go:311
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x23d432a	github.com/go-chi/chi.(*Mux).routeHTTP+0x28a				/source/vendor/github.com/go-chi/chi/mux.go:436
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d69304	code.gitea.io/gitea/routers/common.Middlewares.func2.1+0x84		/source/routers/common/middleware.go:72
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68528	code.gitea.io/gitea/routers/common.LoggerHandler.func1.1+0x368		/source/routers/common/logger.go:23
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d650c7	github.com/go-chi/chi/middleware.StripSlashes.func1+0x107		/source/vendor/github.com/go-chi/chi/middleware/strip.go:30
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d605f3	github.com/chi-middleware/proxy.ForwardedHeaders.func1.1+0xf3		/source/vendor/github.com/chi-middleware/proxy/middleware.go:37
#	0x14756ca	net/http.HandlerFunc.ServeHTTP+0x4a					/usr/local/go/src/net/http/server.go:2049
#	0x2d68e47	code.gitea.io/gitea/routers/common.Middlewares.func1.1+0x67		/source/routers/common/middleware.go:25
#	0x14756ca	

[zeripath]

I think we might need to pass the outputs of the profiles through pprof ourselves.

if you give us the files obtained from:

curl 'http://127.0.0.1:6060/debug/pprof/goroutine?debug=1' -o goroutine.out
curl 'http://127.0.0.1:6060/debug/pprof/profile?seconds=30' -o profile.out
curl 'http://127.0.0.1:6060/debug/pprof/trace?seconds=30' -o traceprofile.out
curl 'http://127.0.0.1:6060/debug/pprof/block?debug=1' -o block.out
curl 'http://127.0.0.1:6060/debug/pprof/mutex?debug=1' -o mutex.out

We may be able to divine where the deadlock is occurring.

---

you may need to zip these up.

[dannyzcy]

Gitea.zip

5 files are in zip file, thanks