Repo-updater floods GHE with API requests

[mike-r-mclaughlin]

• Sourcegraph version: 5.2.1
• Reported by: https://github.com/sourcegraph/accounts/issues/6716

Details

See also: https://github.com/sourcegraph/sourcegraph/issues/60941
Discussion in Slack Thread, recorded troubleshooting session.

repo-updater seems to be continuously sending requests like this:

{"SeverityText":"ERROR","Timestamp":1709120061200289151,"InstrumentationScope":"repo-updater.UpdateScheduler.RunUpdateLoop","Caller":"repos/scheduler.go:191","Function":"github.com/sourcegraph/sourcegraph/internal/repos.(*UpdateScheduler).runUpdateLoop.func1","Body":"error updating repo","Resource":{"service.name":"repo-updater","service.version":"5.2.1","service.instance.id":"fd2b0393e88e"},"Attributes":{"err":"repo git.exmple.com/org1/repo1:: failed to fetch repo \"git.examle.com/org1/repo1\": exit status 128 - output: \"remote: Repository not found.\\nfatal: repository 'https://git.example.com/org1/repo1/' not found\\n\"","uri":"git.example.com/org1/repo1"}}

and

{"SeverityText":"WARN","Timestamp":1709120061632231040,"InstrumentationScope":"repo-updater.PermsSyncer.syncRepoPerms","Caller":"authz/perms_syncer.go:140","Function":"github.com/sourcegraph/sourcegraph/cmd/repo-updater/internal/authz.(*PermsSyncer).syncRepoPerms","Body":"ignoreUnauthorizedAPIError","Resource":{"service.name":"repo-updater","service.version":"5.2.1","service.instance.id":"fd2b0393e88e"},"Attributes":{"repo":{"ID":285579,"name":"git.example.com/org2/repo2","private":true},"error":"list users for repo: request to https://git.example.com/api/v3/repos/org2/repo2/collaborators?page=1&per_page=100 returned status 404: Not Found","suggestion":"GitHub access token user may only have read access to the repository, but needs write for permissions"}}

(among others) to GHE.

In https://github.com/sourcegraph/accounts/issues/6716, we are seeing ~50k requests every 15 minutes (as measured on GHE). When we shut down the repo-updater container, the requests settle to almost 0. Bringing the container back up causes the requests to spike.

All repos we checked were deleted or had permissions changed longer than 3 months ago.

[eseliger]

Do we know if the customer tracks Git and API requests separately or combined? The two log messages are for two different API, one is from git fetches using gits smart HTTP protocol, the other is from perms syncing and using the REST/GQL APIs from GitHub.

The latter can be limited using rateLimit on the code host connection.

[eseliger]

(If all of the answers are in the Chorus, let me know, it's a 2:40h recording and I didn't really know where to start :D)

[mike-r-mclaughlin]

Do we know if the customer tracks Git and API requests separately or combined? The two log messages are for two different API, one is from git fetches using gits smart HTTP protocol, the other is from perms syncing and using the REST/GQL APIs from GitHub.

They are tracked separately. This particular issue is with API calls only. When looking at the actual API calls it seems to be permission syncing related (based on the GitHub log messages)

The latter can be limited using rateLimit on the code host connection.

They had set the rateLimit initially. But, they only set it on prod server so it was only reducing the load some. (more details in the slack thread on this). We can try to set rateLimit again to help slow this down for now but it did seem like the log messages (and therefore the API calls) were happening excessively (just my gut instinct from seeing the logs on both sides).

[mike-r-mclaughlin]

Here is a count of the API calls as seen by the GHE server over a 4 hour period:

The overwhelming majority of calls are for repository permissions.

Details

• We consistently see a 403 followed by 200 for each API request (always in a pair). Presumably it sends an anonymous request first and then responds with the authentication when it gets a 403?
• We also believe GHE is logging 2 requests for each successful RepoPermissions API call. So, the 1.7M requests in 4 hours is actually ~1.15M requests (approximately 2/3 of the total amount because of the 403+200 pair)
• ~100k repositories on this server

After reading this doc, it appears like this may be expected behavior.

It seems like rateLimit might be the only option to control this. Or, perhaps through the use of webhooks