Any events on webhook send twice

[matrozov]

• Gitea version (or commit ref): 1.11.0+dev-331-g055f6d229
• Git version: not needed
• Operating system: Docker container from gitea/gitea
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

I deployed gitea in my kubernetes from the docker image of gitea/gitea. I linked the webhook to the created repository and noticed that all events are fired twice (push, create / delete branch and tags, etc.). The pull-request event was triggered as many as 4 times per action, but there is a separate issue for this #9181. In the images below, I pushed a new brunch and received two push and brunch creation events each. Webhook has been configured in both "All Events" and "Push events" mode. In both cases, push events arrived twice, and in "All events" mode all events came twice.

Screenshots

[lunny]

Have you add webhooks on both repository webhook setting and orgnization webhook setting?

[matrozov]

In the beginning, I used only the settings in the repository. Then, for the test, webhook was also added to the global settings, but this did not affect the double sending in any way. Duplicate notifications were from the start. Subsequent experiments with deleting, adding webhook, repositories and other types of events did not change the situation.

[lunny]

I cannot reproduce this on my local instance. Could you reproduce this on try.gitea.io ?

[matrozov]

Not. I tried, but on try.gitea.io I received only neighboring bugs #9184, #9181 and #9180. I will try to deploy gitea on a separate host again today. Perhaps something is tied to the initial settings.

[matrozov]

I received confirmation that the duplication of webhook depends on the installation configuration. I deployed on the local machine about 10 times and got different results depending on whether I redefined the parameters through ENV. In my product version in kubernetes, where I initially received an error, I also deployed with parameters predefined via ENV.

I attach my docker-compose.yaml file, as well as the resulting app.ini. At the installation stage, I only specify ssh_domain and administrator authorization settings. As far as I could understand, if you do not pass EMV when deploying, then gitea does not duplicate webhook. I did not have enough time to figure out more specifically which parameter in the deployment leads to such an error. I hope you can repeat my experiment.

test3.zip

After the deployment and installation, I did:

• added ssh key to user profile
• created a new repository
• added a webhook to this repository with all the default parameters (specified only url)
• cloned the repository locally (via ssh)
• did push

[stklcode]

I experience the same issue. Every webhook (types "Slack" and "Gitea" in use, all added at repository level) is triggered twice when pushing from Git client (SSH or HTTPS). As far as I can tell, it happens on any repository.
Same JSON body (except for the timestamp +/- 1 second), same event-header, different delivery-header.

I'm running two separate instances. Both same version and about the same configuration, updated regularily since 1.1 or 1.2. Sadly I cannot remember when it started.

Faulty instance:
Version: 1.10.1
Platform: Linux, amd64 (Debian Stretch, Kernel 3.16)
DB: MySQL (MariaDB 10.1)
Deployment: Direct setup (Downloaded from https://dl.gitea.io)
Configuration:

app.ini

APP_NAME = git @ mydomain.example.com
RUN_USER = git
RUN_MODE = prod

[database]
DB_TYPE  = mysql
HOST     = /var/run/mysqld/mysqld.sock
NAME     = gitea
USER     = gitea
PASSWD   = ***
SSL_MODE = disable
PATH     = data/gitea.db

[repository]
ROOT = /opt/gitea/repo

[server]
SSH_DOMAIN             = git.mydomain.example.com
PROTOCOL               = unix
HTTP_ADDR              = /opt/gitea/gitea.sock
HTTP_PORT              = 3000
UNIX_SOCKET_PERMISSION = 666
ROOT_URL               = https://git.mydomain.example.com/
DISABLE_SSH            = false
DISABLE_HTTP_GIT       = true
SSH_PORT               = 2222
SSH_LISTEN_PORT        = 2222
START_SSH_SERVER       = true
OFFLINE_MODE           = true
LANDING_PAGE           = explore

[mailer]
ENABLED = true
HOST    = mail.example.com:587
FROM    = "Git" <git@example.com>
USER    = git@example.com
PASSWD  = ***

[service]
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL     = true
DISABLE_REGISTRATION   = true
ENABLE_CAPTCHA         = false
REQUIRE_SIGNIN_VIEW    = false

[picture]
DISABLE_GRAVATAR        = true
ENABLE_FEDERATED_AVATAR = false

[session]
PROVIDER      = file
COOKIE_SECURE = true
COOKIE_NAME   = git_session

[log]
MODE      = file
LEVEL     = Info
ROOT_PATH = /opt/gitea/log

[security]
INSTALL_LOCK       = true
SECRET_KEY         = ***
INTERNAL_TOKEN     = ***
IMPORT_LOCAL_PATHS = false

[other]
SHOW_FOOTER_BRANDING           = false
SHOW_FOOTER_VERSION            = false
SHOW_FOOTER_TEMPLATE_LOAD_TIME = false

[oauth2]
JWT_SECRET = ***

Working instance:
Version: 1.10.1
Platform: Linux, amd64 (Debian Buster, Kernel 4.19)
Deployment: Docker Stack (official Image gitea/gitea:1.10, only USER_UID, USER_GID env)
DB: MySQL (MariaDB 10.3)
Configuration:

app.ini

APP_NAME = git @ otherdomain.example.com
RUN_MODE = prod

[repository]
ROOT = /data/git/repositories

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH   = /data/gitea
SSH_DOMAIN      = otherdomain.example.com
HTTP_PORT       = 3000
ROOT_URL        = https://otherdomain.example.com/
DISABLE_SSH     = false
SSH_PORT        = 2222
SSH_LISTEN_PORT = 22

[database]
DB_TYPE  = mysql
HOST     = /var/run/mysqld/mysqld.sock
NAME     = gitea
USER     = gitea
PASSWD   = ***

[session]
PROVIDER_CONFIG = /data/gitea/sessions

[mailer]
ENABLED = false

[picture]
AVATAR_UPLOAD_PATH      = /data/gitea/avatars
DISABLE_GRAVATAR        = true
ENABLE_FEDERATED_AVATAR = false

[log]
MODE      = file
LEVEL     = Warn
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK   = true
SECRET_KEY     = ***
INTERNAL_TOKEN = ***

[other]
SHOW_FOOTER_BRANDING           = false
SHOW_FOOTER_VERSION            = false
SHOW_FOOTER_TEMPLATE_LOAD_TIME = false

[oauth2]
JWT_SECRET = ***

[stklcode]

I've done some additional testing on the first of the above instances.

Two repositories (real world, no test dummies), both 1 Slack webhook and 1 Gitea webhook. Same configuration, same owner, same commit (dummy file on new temp-branch).

One repository (the slightly older one) behaves correcly, the other triggers twice. Both logs (Trace level) attached:
https://gist.github.com/stklcode/0e09e79033173413d55b9f800916a7e3

I cannot debug in that instance, as it is live and prod, but I had a look into the code.
DeliverHooks() iterates over undelivered tasks twice. For some reason the hooks appear in both selects, i.e. they are either places placed twice in the DB before or not marked as delivered.
deliver.go in master:

gitea/modules/webhook/deliver.go

Line 152 in 2b0e4e0

func DeliverHooks(ctx context.Context) {

webhooks.go in v1.10:

gitea/models/webhook.go

Line 930 in f0f48e0

func DeliverHooks() {

[lunny]

I take a look at two implementation on master and v1.10. It's the same.

[stklcode]

I've updated from 1.10.1 to 1.10.2 today. No change.

However by chance I've found the origin of the bug, because I have migrated my Gitea to a different directory structure and had to re-synchronize all hook scripts.

Apparently there are 2 identical hook files for pre- and post-receive phase, one named gitea (the correct one) and one named like the phase.

<repo>/hooks/
 |- pre-receive.d/
 |-- gitea
 |-- pre-receive
 |- post-receive.d/
 |-- gitea
 |-- post-receive

(not for update)

Last modification of the "wrong" scripts is somewhere back in 2017, the gitea scripts are modified today (I've triggered the "synchronize ... hooks for all repositories" task).

So the hook is really triggered twice externally.

[lunny]

pre-receive.d/gitea is managed by gitea internal. pre-receive.d/pre-receive could be changed on gitea UI if you have enough permission.

[stklcode]

I know. Content was exactly the same for all affected repos and I‘m pretty sure I never configured this and none of the collaborators knew the binary/config paths... 🤔

[lunny]

If the content in pre-receive.d/pre-receive is the same as which in pre-receive.d/gitea.

[theAkito]

However by chance I've found the origin of the bug, because I have migrated my Gitea to a different directory structure and had to re-synchronize all hook scripts.

Same here and same issue.
This bug is 100% confirmed.

My Drone is going crazy and begging for mercy...