Using `exec -a` in `gitea/gitea:dev` container broke SSH passthrough access

[Myridium]

Gitea Version

Gitea version 1.16.0

How are you running Gitea?

gitea/gitea:dev docker.

Description

Gitea Docker now uses exec -a "$0" ... in /usr/local/bin/gitea, so AppPath is resolved to /usr/local/bin/gitea now.

Before, the AppPath is /app/gitea/gitea.

The AppPath is used inside authorized_keys file and shared on both container and host side.

This can introduce some problems:

1. The document for SSH pass-through are still using /app/gitea/gitea, which misleads users. https://docs.gitea.io/en-us/install-with-docker/#ssh-container-passthrough
2. It breaks existing Docker pass-through users.

The easy fix is using exec .... instead of exec -a "$0" ... (Is there any Pros of using exec -a?)

Or Gitea should update the document and mark this behavior as breaking.

by @wxiaoguang: personally I prefer to keep 1.16 behavior, otherwise it's too strange to force users creating a /app/gitea/ on their host.

I solved this issue (at least temporarily?) in the midst of writing a bug report. But it was caused by an (unknown) bug in gitea which may or may not be resolved. I'm going to leave this bug report here because the cause is unknown to me and it could happen again (e.g. when adding a key) and in case it helps others troubleshoot similar problems in the future.

The short of it: some update to the gitea/gitea:dev container caused the git user's authorized_keys file to rewrite /app/gitea/gitea to /usr/local/bin/gitea, after I pulled the gitea/gitea:dev container for the first time in about a month or 2, and relaunched the image via docker-compose.yml. I am reasonably sure that the gitea 'database version' made a jump from 189 to 205 when I did this (maybe that is helpful information).

The issue

Last time I pulled the gitea/gitea:dev container, it was somewhat ahead of 1.15.8. I pulled the dev version because it contained an unmerged bug fix regarding FIDO and 2FA.

Since pulling the gitea/gitea:dev container again yesterday (I forgot I had the dev tag set), I can no longer access the SSH passthrough, with this error:

local-pc $ git push
> bash: line 1: /usr/local/bin/gitea: No such file or directory
> fatal: Could not read from remote repository.
>
> Please make sure you have the correct access rights
> and the repository exists.

Troubleshooting and resolving

This error makes no sense to me, as I run on the remote server::

remote-server $ docker exec -it container_gitea bash
bash-5.1# ls -lahF /usr/local/bin
> total 10M    
> drwxr-xr-x    1 root     root        4.0K Dec 27 02:52 ./
> drwxr-xr-x    1 root     root        4.0K Dec 27 02:49 ../
> -rwxr-xr-x    1 root     root        9.4M Dec 27 02:52 environment-to-ini*
> -rwxr-xr-x    1 root     root         582 Dec 27 02:49 gitea*

Showing that /usr/local/bin/gitea is readable and executable by everyone.

Gitea is configured to run with a git user like so in the docker-compose.yml:

  server:
    # Latest version as of now still includes a bug that doesn't allow FIDO enrolment unless 2FA is enabled.
    #image: gitea/gitea:1.15.8
    image: gitea/gitea:dev
    container_name: container_gitea
    environment:
      - USER_UID=${GIT_USER_UID?No GIT_USER_UID set in env file}
      - USER_GID=${GIT_USER_GID?No GIT_USER_GID set in env file}
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=db:5432
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=gitea
      - GITEA__database__PASSWD=gitea

Naturally, I try downgrading to a previous version of gitea. I think that my old gitea/gitea:dev image before pulling was compatible with 1.15.8 (I left that comment you see in the docker-compose.yml). So I try reverting to that version. gitea fails to start however, and the docker logs container_gitea shows:

2021/12/29 12:42:17 ...les/storage/local.go:47:NewLocalStorage() [I] Creating new Local Storage at /data/gitea/repo-archive
2021/12/29 12:42:17 routers/init.go:93:GlobalInit() [I] SQLite3 Supported
2021/12/29 12:42:17 routers/common/db.go:20:InitDBEngine() [I] Beginning ORM engine initialization.
2021/12/29 12:42:17 routers/common/db.go:27:InitDBEngine() [I] ORM engine initialization attempt #1/10...
2021/12/29 12:42:17 ...om/urfave/cli/app.go:524:HandleAction() [I] PING DATABASE postgres
Downgrading database version from '205' to '189' is not supported and may result in loss of data integrity.
If you really know what you're doing, execute `UPDATE version SET version=189 WHERE id=1;`
2021/12/29 12:42:17 ...ations/migrations.go:414:Migrate() [F] Downgrading database version from '205' to '189' is not supported and may result in loss of data integrity.
        If you really know what you're doing, execute `UPDATE version SET version=189 WHERE id=1;`
Received signal 15; terminating.

Uh oh. I'm not messing around with a database downgrade, especially when the version difference looks so enormous. So I'm going to have to try to rectify this issue with the container I have. (I do docker tag gitea/gitea:dev gitea/gitea:peg and change the docker-compose.yml to use this. I don't want to be on dev anymore and when a new compatible fixed version tag becomes available, e.g. gitea/gitea:1.16, I will use that.)

Now I try simply sshing into the server to see what happens:

local-pc $ ssh git@remote-server
PTY allocation request failed on channel 0
bash: line 1: /usr/local/bin/gitea: No such file or directory
Connection to <hostname> closed.

So now I docker stop the container_gitea entirely, and I try sshing again and get the same result as above! Okay, so something wrong with the SSH passthrough configuration then?

I check and see that /app/gitea/gitea on remote-server is readable and executable by everything (as is /app and /app/gitea). Yes, that's fine.

Then I try sshing using a local user:

remote-pc $ ssh git@localhost
PTY allocation request failed on channel 0
bash: line 1: /usr/local/bin/gitea: No such file or directory
Connection to <hostname> closed.

Then with the git user which gitea runs as: this works.

git@remote-pc $ ssh git@localhost
> Last login: ....
git@remote-pc $ 

I don't understand. Here are the contents of /app/gitea/gitea:

#!/bin/sh
# Taken from https://docs.gitea.io/en-us/install-with-docker/#ssh-container-passthrough
ssh -p 2221 -o StrictHostKeyChecking=no git@127.0.0.1 "SSH_ORIGINAL_COMMAND=\"$SSH_ORIGINAL_COMMAND\" $0 $@"

So now I check remote-server:/home/git/.ssh/authorized_keys; this is where the magic is supposed to happen which forwards requests to gitea. Bingo. Here's the problem:

# gitea public key
command="/usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-12",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,restrict ssh-rsa <user1_SSH_PUB_KEY> user1@localhost
# gitea public key
command="/usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-10",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,restrict ssh-rsa <some_SSH_PUB_KEY>
# SSH pubkey from the Host machine's git user

ssh-rsa <gituser_SSH_PUB_KEY> Git Host Key

# Other keys from users of Gitea
command="/home/git/forward_ssh_to_gitea.sh --config=/data/gitea/conf/app.ini serv key-1",no-port-forward,no-X11-forwarding,no-agent-forwarding,no-pty <user pubkey>

I only ever added keys to gitea using the web interface, and I didn't do anything like ssh-copy-id git@remote-server to copy keys manually to that user.

Thankfully gitea seems to keep backups of authorized_keys. I copied over one of the two most recent backups (same minute) sitting in the same directory next to authorized_keys. It looks like this:

# gitea public key
command="/app/gitea/gitea --config=/data/gitea/conf/app.ini serv key-10",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,restrict ssh-rsa <some_SSH_PUB_KEY> 
# gitea public key
command="/app/gitea/gitea --config=/data/gitea/conf/app.ini serv key-12",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,restrict ssh-rsa <user1_SSH_PUB_KEY> user1@localhost
# SSH pubkey from the Host machine's git user

ssh-rsa <gituser_SSH_PUB_KEY> Git Host Key

# Other keys from users of Gitea
command="/home/git/forward_ssh_to_gitea.sh --config=/data/gitea/conf/app.ini serv key-1",no-port-forward,no-X11-forwarding,no-agent-forwarding,no-pty <user pubkey>

After replacing authorized_keys with a copy of this backup, ssh and git <command> git@remote-server are now working.

[wxiaoguang]

Can you check what happens to your /usr/local/bin/gitea? I believe the key point is the message you saw bash: line 1: /usr/local/bin/gitea: No such file or directory. However according to the Gitea Dockerfile, the /usr/local/bin/gitea should be there.

[Myridium]

Can you check what happens to your /usr/local/bin/gitea? I believe the key point is the message you saw bash: line 1: /usr/local/bin/gitea: No such file or directory. However according to the Gitea Dockerfile, the /usr/local/bin/gitea should be there.

This is using SSH passthrough, so /home/git/.ssh/ is on remote-server, not inside the container. It should not be looking for /usr/local/bin/gitea; it should be executing /app/gitea/gitea. See https://docs.gitea.io/en-us/install-with-docker/#ssh-container-passthrough

The docs for instance say that authorized_keys should be

# SSH pubkey from git user
ssh-rsa <Gitea Host Key>

# other keys from users
command="/app/gitea/gitea --config=/data/gitea/conf/app.ini serv key-1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty <user pubkey>

The /app/gitea/gitea is where we have to place our executable. I'm not sure exactly what this line does, but new authorized_keys are supposed to be added according to that template. As can be seen above, they spontaneously changed themselves to break the template, due to an update.

[wxiaoguang]

@zeripath I updated the comment, how do you think about these two solutions?

[Myridium]

Can we have a way to set our own path to the executable instead of /app/gitea/gitea? The fixed path is problematic. For me it's okay, I have root access to the machine and I created the directory /app/gitea/ and popped in a symlink at /app/gitea/gitea. But generally speaking, users may not have access to whatever hardcoded path you require them to use.

I also don't like cluttering up /. Docker applications should be as portable as possible. Having to go through these extra hoops is a bit of a headache. But I understand that something like this is probably necessary for SSH passthrough on port 22.

[wxiaoguang]

That's a more complex design problem.

Now, the SSH pass-through uses (shares) Gitea's authorized_keys file on host side, that's the key problem. If you can separate the two authorized_keys files (eg: AuthorizedKeysCommand), then the problem could be resolved.

[Myridium]

A combined authorized_keys is necessary because of the way ssh (through git) works. For a standard setup where you write git clone git@XYZ:user/project.git, you need the git user to be accessible via ssh on port 22 at the remote server. So, users who are validated to gitea need to also be validated to the git user on the remote machine. So, the authorized_keys file must be kept in sync. Also, users may access the git repositories directly through their path on disk instead of through gitea. I don't think this can or should change.

The real issue I think is why there is an executable required at /app/gitea/gitea at all.

[wxiaoguang]

There are two /app/gitea/gitea files:

One is the bridge script on your host, it just executes the real gitea app inside docker.
Another is the real gitea app inside docker.

However, the guide only uses one shared authorized_keys, there is only one path for both gitea executables.

user -> git@host -> read authorized_keys -> command gitea app (bridge) -> git@docker -> read authorized_keys -> command gitea app (real)

[Myridium]

There are two /app/gitea/gitea files:

One is the bridge script on your host, it just executes the real gitea app inside docker. Another is the real gitea app inside docker.

However, the guide only uses one shared authorized_keys, there is only one path for both gitea executables.

user -> git@host -> read authorized_keys -> command gitea app (bridge) -> git@docker -> read authorized_keys -> command gitea app (real)

Thanks for clarifying.

In that case, gitea should not require a file on the host machine. To require manual setup like this goes against the spirit of dockerisation and portability. The extra binary /app/gitea/gitea is unecessary and can be baked into the command option of the ssh.

Working example:

[root@remote-server .ssh]# cat authorized_keys
# gitea public key
command="GITEA_KEYNUM=10 ; GITEA_BIN=\"/app/gitea/gitea\" ; SSH_PASSTHROUGH_PORT=\"2221\" ; passthrough_required() { ! [ -f \"$GITEA_BIN\" ] ; } ; gitea_ssh_command() { passthrough_required || { \"$GITEA_BIN\" \"$@\" ; return $? ; } ; ssh -p \"$SSH_PASSTHROUGH_PORT\" -o StrictHostKeyChecking=no git@127.0.0.1 \"SSH_ORIGINAL_COMMAND=\\"$SSH_ORIGINAL_COMMAND\\" \\"$GITEA_BIN\\" $@\" ; } ; gitea_ssh_command --config=/data/gitea/conf/app.ini serv \"key-$GITEA_KEYNUM\"",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,restrict ssh-rsa <SOME_SSH_PUB_KEY> [user@hostname]

# gitea public key
command="GITEA_KEYNUM=12 ; GITEA_BIN=\"/app/gitea/gitea\" ; SSH_PASSTHROUGH_PORT=\"2221\" ; passthrough_required() { ! [ -f \"$GITEA_BIN\" ] ; } ; gitea_ssh_command() { passthrough_required || { \"$GITEA_BIN\" \"$@\" ; return $? ; } ; ssh -p \"$SSH_PASSTHROUGH_PORT\" -o StrictHostKeyChecking=no git@127.0.0.1 \"SSH_ORIGINAL_COMMAND=\\"$SSH_ORIGINAL_COMMAND\\" \\"$GITEA_BIN\\" $@\" ; } ; gitea_ssh_command --config=/data/gitea/conf/app.ini serv \"key-$GITEA_KEYNUM\"",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,restrict ssh-rsa <user1_SSH_PUB_KEY> user1@localhost
# SSH pubkey from the Host machine's git user

ssh-rsa <gituser_SSH_PUB_KEY> Git Host Key

# Other keys from users of Gitea
command="GITEA_KEYNUM=1 ; GITEA_BIN=\"/app/gitea/gitea\" ; SSH_PASSTHROUGH_PORT=\"2221\" ; passthrough_required() { ! [ -f \"$GITEA_BIN\" ] ; } ; gitea_ssh_command() { passthrough_required || { \"$GITEA_BIN\" \"$@\" ; return $? ; } ; ssh -p \"$SSH_PASSTHROUGH_PORT\" -o StrictHostKeyChecking=no git@127.0.0.1 \"SSH_ORIGINAL_COMMAND=\\"$SSH_ORIGINAL_COMMAND\\" \\"$GITEA_BIN\\" $@\" ; } ; gitea_ssh_command --config=/data/gitea/conf/app.ini serv \"key-$GITEA_KEYNUM\"",no-port-forward,no-X11-forwarding,no-agent-forwarding,no-pty <user pubkey>

and so on.

FYI:
I could not find any information on acceptable string formatting inside authorized_keys. E.g. it is not possible to insert line breaks with a backslash \. This IBM documentation suggests that everything is interpreted literally inside the quotes, except quotes themselves, which need a backslash in front of them. (probably also backslashes can be escaped with another backslash).
I inferred from an error message that the command string is issued to bash -c.

I made it more verbose so you can see how different parts can be edited, particularly GITEA_KEYNUM, GITEA_BIN, SSH_PASSTHROUGH_PORT, the bash function passthrough_required which can be made more robust than checking whether GITEA_BIN exists (e.g. check for UUID of the target gitea instance or something).

[zeripath]

Gitea is not the reason for requiring anything on the host - your decision not to pass every ssh request directly to Gitea is the reason you're needing to consider this.

SSH is not the same as HTTP.

So you have to make some decisions about how you're going to arrange communication between your host ssh and the Gitea ssh.

This is not our fault and claiming that something isn't the docker way isn't helpful. As far as I have been able to find openssh and the ssh protocol is not built to allow routing like you want it. Especially not the way it has to be used for git.

Gitea did not invent the use of the common git user for ssh. Git didn't do that either. Having a common user using authorized_keys to identify the real user dates back to at least the SVN days but likely CVS days too. This precedes docker by some decades.

One suggestion which people found simplest was to use and munge the authorized_keys file so that was suggested, however Gitea has had a keys command which fits with the authorized keys command output. Others have found that a script that calls docker exec works better. You may want to try that instead.

However moaning about how X or Y isn't the docker way is NOT going to induce me to spend any time thinking about your problem.

In your case simply just add a symbolic link or replace the /usr/local/bin/gitea with what you need it to be in the keys file using a regular expression.

[Myridium]

Gitea is not the reason for requiring anything on the host - your decision not to pass every ssh request directly to Gitea is the reason you're needing to consider this.

SSH is not the same as HTTP.

So you have to make some decisions about how you're going to arrange communication between your host ssh and the Gitea ssh.

This is not our fault and claiming that something isn't the docker way isn't helpful. As far as I have been able to find openssh and the ssh protocol is not built to allow routing like you want it. Especially not the way it has to be used for git.

Gitea did not invent the use of the common git user for ssh. Git didn't do that either. Having a common user using authorized_keys to identify the real user dates back to at least the SVN days but likely CVS days too. This precedes docker by some decades.

One suggestion which people found simplest was to use and munge the authorized_keys file so that was suggested, however Gitea has had a keys command which fits with the authorized keys command output. Others have found that a script that calls docker exec works better. You may want to try that instead.

However moaning about how X or Y isn't the docker way is NOT going to induce me to spend any time thinking about your problem.

In your case simply just add a symbolic link or replace the /usr/local/bin/gitea with what you need it to be in the keys file using a regular expression.

You don't seem to understand this thread.

I posted a solution which sacrifices nothing and solves the issue. There is little to think about. Tweak a bit and merge. Issue resolved. I can't help it if you choose to be stubborn or if you do not understand this thread.

The Gitea documentation explicitly supports SSH passthrough. A commitment has been made to support SSH passthrough. You cannot now moan when users expect SSH passthrough not to break.

Also try removing support for SSH passthrough, see very quickly just how many people will 'moan'. I'm not sure if you've ever used Gitea docker before, because it becomes very clear why SSH passthrough is needed once you have.

[jpraet]

Upgrade to 1.16.0-rc1 broke the docker SSH passthrough for me as well.
I worked around it with ln -s /app/gitea/gitea /usr/local/bin/gitea on the docker host.

[christian-heusel]

Thanks for the fix @jpraet! 🥳 I think this should either be fixed or be mentioned in the release notes!

[Myridium]

Thanks for the fix @jpraet! 🥳 I think this should either be fixed or be mentioned in the release notes!

As @jpraet said, it is a workaround, not a fix. I posted a better fix above that doesn't require a file on the host.