cron.update_mirrors broken in Gitea 1.16.0

[somera]

Gitea Version

1.16.0

Git Version

2.25.1

Operating System

Ubuntu 20.04.3

How are you running Gitea?

Precompiled gitea-1.16.0-linux-amd64

Database

PostgreSQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

I'm hosting a lot of mirrors on my Gitea instance. Some days ago I updated to 1.16.0.

I see -> it started at 04:00 AM, but ...

Last run with Gitea 1.15.x:

cron.update_mirrors needed with Gitea 1.15.x ~3h to update all mirrors. The cron runs fridays at 04:00 AM. And today was the first run.

I saw in gitea that the cron runs -> run count was 1. But the monitoring data shows no activity (RAM usage, CPU usage, Disk usage) between 04:00 and 07:00 AM.

Than I started (4-6x times) the cron in Gitea admin web console. It runs for some seconds. and I saw some activity.

But there is no date for the last run of the cron

After the 4-6 runs the sorting by last update time isn't showing any changes

All the projects with Feb 04 date are created today. But I can't see the updated mirrors when I change the sort order.

Why cron.update_mirrors is not updating all my mirrors? I can't see any errors in the Gitea logs.

Screenshots

No response

[somera]

@zeripath should I set now the new parameters from #16982?

Cause I added now

PULL_LIMIT = 100

and it runs longer.

After an update (1.15.x -> 1.16.0) I rather expect the old behavior.

[somera]

I increased it to

PULL_LIMIT = 2000

and the cron.update_mirrors run's ~8-10 minutes. Second time ~12 minutes. Means for me: 2 runns -> 4000 updated 2000 mirrors.

but ~20 minutes for 4000 mirrors it's to fast for me.

Questions:

• I'm right or wrong?
• Are there some optimizations? If yes -> which?

Sometimes I see what Gitea is doing

and sometimes not

Just missing the git call for a lot projects.

And it will be good to know that this cron was running (start date, finish date) and how many repos was updated.

You are showing some infos for some crons. But not for the mirror cron.

I just have to know: is the process working fine or not? Can I lost data?

[somera]

I run the cron 4x manually.

1 time with PULL_LIMIT = 100 and 3 times with PULL_LIMIT = 2000

And in the DB I see

I expected ~6100 repos with update_unix = 2022-02-04. But there are only 1675.

[somera]

Now I'm sure. The new process is broken.

6 runs: 1 time with PULL_LIMIT = 100 and 5 times with PULL_LIMIT = 2000

But

And if I start the cron again -> nothing happens.

And in the mirror table (next_update_unix column) I see, how many mirrors can be updated

[somera]

Since yesterday klicked x times on "Now sync" -> nothing happens

[somera]

I run today manually the cron. Gitea updated run some mirror updates. But the mirror.updated_unix column hasn't after the run ZERO changes.

Now I call the curl

curl -X 'POST' 'http://nuc-mini-celeron:3000/api/v1/repos/gaphor/in-app-notification-demo/mirror-sync' -H 'accept: application/json' -H 'Authorization: token xxxxx' -d ''
call for evry mirror again and the mirror.updated_unix is changing

But the whole mirror update process looks brocken.

[somera]

After spending some hours with Gitea 1.16.0 I'm disapointed. Disapointed cause Gitea is not working like in 1.15.x for mirror sync. Yes, there was changes. But the changes shouldn't change the existing behavior.

I sended ~40.000 curl calls (4x for all mirrors) to get this mirror_update status

1785 mirrors are not up2date.

This is my current config:

[repository]
ROOT = /var/lib/gitea/repositories
; Mirror sync queue length, increase if mirror syncing starts hanging
; DEPRECATED!!!
MIRROR_QUEUE_LENGTH = 20000

[mirror]
DEFAULT_INTERVAL = 8h
LENGTH = 20000

; Update mirrors
[cron.update_mirrors]
SCHEDULE = 0 0 4 * * 5 ; -> on 04.02.2022 04:00:00 it runs only for 8-10 minutes
PULL_LIMIT = -1
PUSH_LIMIT = -1

If I now start the update_mirrors cron manually, the oldest mirrors don't be updated. Nothing happens. Why?

If I open the repo with the oldest sync date

and I click on sync. It's not syncing. Why not? How long have I to wait?

[somera]

Same status after the very short cron run today at 04:00AM

[lunny]

Are there any logs?

[somera]

Are there any logs?

Hm ... if I switch TO DEUBUG I see a lot of SQL's.

Or this: #16982 (comment)

I think, the "problem" is the new feature implemented in #16982. Or you did other big changes.

[somera]

I added

[queue.mirror]
LENGTH = 20000

and than

./gitea manager flush-queues

and restart gitea. After I run mirror-cron manually I see no changes.

And I can bet, that if I would do my ~9000 curl calls, some of the mirrors will be synced.

The 1 repo with mirror.updated_unix = 2022-02-06 is a new repo which I added today.

[zeripath]

I've held off replying to this issue because I'm struggling to completely understand what is the problem and what is actually happening.

This is despite you having posted 11 comments...

So can we please have a succinct three line description as to what is happening, what you want to happen and what your configuration is? (that includes cron configuration.)

---

I will also remind you that you SPECIFICALLY asked for LIMIT_SIZE - I checked with you as to how it was supposed to work and I wrote a very long explanation as to what it was doing.

There have been many changes to mirroring this is not the only thing.