On Prem Azure Devops Migration fails

[bjornbouetsmith]

Hi,
Any plans to support migration from on prem Azure DevOps (git) to gitea?

I have tried and it just fails - possibly because authentication in azure devops is not using basic authentication but something else?

It just logs:

May 19 18:15:27 git gitea[687]: #033[36m2022/05/19 18:15:27 #033[0m#033[32m...ervices/task/task.go:56:#033[32mhandle()#033[0m #033[1;31m[E]#033[0m Run task failed: #033[1mAuthentication failed: Clone: exit status 128 - fatal: Authentication failed for 'http://devops.root.dom/ROOT/common/_git/Helm/'

I am 100% certain that the username/password combo is correct, since I use it on a daily basis - and also can access the repo from another service - using the exact same combo.

I also have this in my app.ini

[migrations]
ALLOW_LOCALNETWORKS = true

My guess is that it is probably because I have a "special" character in my password and when I turn on debug logging I can see the url it tries to clone - and if I copy/paste that URL and do a manual git clone from the prompt - I also get the same:

fatal: Authentication failed for 'http://devops.root.dom/ROOT/common/_git/Helm/'

If I do a

git clone http://root%5Cbbs@devops.root.dom/ROOT/common/_git/Helm

And manually type in the password - it just works.

I have no idea if its some kind of escaping that happens that messes up the http request inside git - or if its simply wrongly escaped from gitea's side.

Do I just have to accept this?

According to "people" on the internet, then it seems to have been escaped correctly according to https://en.wikipedia.org/wiki/Percent-encoding - but no dice - I have used other systems in the past that did not like my password.

To give a hint about what character that messes it up - its # - which I know is bad in a url - but I refuse to change my password just because some systems do not like it.

wget on the other hand just accepts the encoded url - so I am beginning to think its a bug in git.

Edit:

Just tested on a windows machine and git clone just works with the encoded url (git version 2.33) - on the gitea server (Rocky Linux) - git version 2.27.0 it fails - so either its a bug that is fixed in 2.33 - or an OS issue that is causing this.

Edit2:
Updated git to version 2.31 on the server - and its the same issue. So I am thinking its a git on linux issue.

Originally posted by @bjornbouetsmith in #8689 (comment)

[bjornbouetsmith]

Also tested on debian 11 with git version 2.34.1 and its the same issue.

So git on Linux in my very simple test does simply not handle escaping of characters correctly,

[bjornbouetsmith]

Tested with a different username and a plain password without any special characters in it - and its the same issue. I think git does not support proper http authentication as such, i.e. challenge and response.

When I use the same password with wget
It writes out:

Resolving devops.root.dom (devops.root.dom)... 192.168.1.230
Connecting to devops.root.dom (devops.root.dom)|192.168.1.230|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: NTLM
Connecting to devops.root.dom (devops.root.dom)|192.168.1.230|:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized

Then wget properly encodes a response with the username/password the and request suceeds.

So possibly its a git on linux vs azure devops on prem incompatibility - and what might need to be done instead is use ssh for git cloning, since that would fix the issue entirely.
Then whoever wants to migrate a azure devops repository would need to enable ssh on the server, create an authentication token and then it should just work if gitea pipes the git clone with a ssh url a ssh "url".

[bjornbouetsmith]

I can verify that adding a ssh public key to azure devops allows me to git clone.
So I think what needs to be done for "Azure Devops" in gitea is that it gets its own "migration" type and then tell people to use ssh.

Basically I would guess that what needs to be done is

• User running gitea needs ssh-keys generated
• public key needs to be configured in azure devops
• migrate/clone url should only accept ssh

Success.

If I knew how to program in GO and my UI skills weren't shit - I would create a pull request for this.

[6543]

related: #1635

[KN4CK3R]

Is this still an issue? I'v tested it with a pull and push mirror and both worked.

[bjornbouetsmith]

Is this still an issue? I'v tested it with a pull and push mirror and both worked.

Yes, I just tried to fire up my gitea installation - updated to 17.3 and same exact issue.

Authentication error as I described above.

What type of migration did you try to do?

And please provide exact steps, was it http/https or other url you used?

[KN4CK3R]

@kdumontnu asked me about this issue, so I registered at Devops and created a repo. The Devops UI told me to use admin as username and zpv5hppqgmk5p76b...h4bwrrpaq as password. Then I setup a push mirror from Gitea to Devops with the provided url and credentials. Gitea created a mirror information like this in the git config file:

[remote "remote_mirror_CDjCL4odYe"]
    url = https://admin:zpv5hppqgmk5p76b...h4bwrrpaq@dev.azure.com/kn4ck3r/test/_git/test
    mirror = true
    push = +refs/heads/*:refs/heads/*
    push = +refs/tags/*:refs/tags/*

That resulted in

After that I created a (mirror) migration for that repo and that worked too.

[bjornbouetsmith]

[remote "remote_mirror_CDjCL4odYe"]
url = https://admin:zpv5hppqgmk5p76b...h4bwrrpaq@dev.azure.com/kn4ck3r/test/_git/test
mirror = true
push = +refs/heads/:refs/heads/
push = +refs/tags/:refs/tags/

You are using azure devops "cloud version" - this issue is about on prem dev ops.

Also - I am not trying to set up a mirror - simply migrate a git repository from on prem azure devops to gitea.

[KN4CK3R]

Oh, sorry about that. I will try with an on prem dev ops.

[bjornbouetsmith]

Oh, sorry about that. I will try with an on prem dev ops.

No worries - but make sure you try a migration - and not try to set up any mirrors.

[KN4CK3R]

Got it working. Basically the on prem Devops needs the Authorization header for cloning. https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops&tabs=Linux#use-a-pat

I will not create a PR for that because this should be implemented as a generic "add custom headers to git requests" instead of just the Authorization header. A possible implementation needs to support the migration/clone, pull/push mirror, LFS client pull/push, ... and everything I forgot. It may not be worth the effort just to enable migrations from on prem Devops as no other service needs this.

These are the code changes I made:

diff --git a/modules/git/repo.go b/modules/git/repo.go
index 8ba3ae4fd..f601b7300 100644
--- a/modules/git/repo.go
+++ b/modules/git/repo.go
@@ -98,16 +98,17 @@ func (repo *Repository) IsEmpty() (bool, error) {
 
 // CloneRepoOptions options when clone a repository
 type CloneRepoOptions struct {
-       Timeout       time.Duration
-       Mirror        bool
-       Bare          bool
-       Quiet         bool
-       Branch        string
-       Shared        bool
-       NoCheckout    bool
-       Depth         int
-       Filter        string
-       SkipTLSVerify bool
+       Timeout             time.Duration
+       Mirror              bool
+       Bare                bool
+       Quiet               bool
+       Branch              string
+       Shared              bool
+       NoCheckout          bool
+       Depth               int
+       Filter              string
+       SkipTLSVerify       bool
+       AuthorizationHeader string
 }
 
 // Clone clones original repository to target path.
@@ -126,6 +127,9 @@ func CloneWithArgs(ctx context.Context, args []CmdArg, from, to string, opts Clo
        if opts.SkipTLSVerify {
                cmd.AddArguments("-c", "http.sslVerify=false")
        }
+       if opts.AuthorizationHeader != "" {
+               cmd.AddArguments("-c").AddDynamicArguments("http.extraHeader=" + opts.AuthorizationHeader)
+       }
        if opts.Mirror {
                cmd.AddArguments("--mirror")
        }
diff --git a/modules/migration/options.go b/modules/migration/options.go
index 1e92a1b0b..944d83bc1 100644
--- a/modules/migration/options.go
+++ b/modules/migration/options.go
@@ -11,13 +11,15 @@ import "code.gitea.io/gitea/modules/structs"
 // this is for internal usage by migrations module and func who interact with it
 type MigrateOptions struct {
        // required: true
-       CloneAddr             string `json:"clone_addr" binding:"Required"`
-       CloneAddrEncrypted    string `json:"clone_addr_encrypted,omitempty"`
-       AuthUsername          string `json:"auth_username"`
-       AuthPassword          string `json:"-"`
-       AuthPasswordEncrypted string `json:"auth_password_encrypted,omitempty"`
-       AuthToken             string `json:"-"`
-       AuthTokenEncrypted    string `json:"auth_token_encrypted,omitempty"`
+       CloneAddr              string `json:"clone_addr" binding:"Required"`
+       CloneAddrEncrypted     string `json:"clone_addr_encrypted,omitempty"`
+       AuthUsername           string `json:"auth_username"`
+       AuthPassword           string `json:"-"`
+       AuthPasswordEncrypted  string `json:"auth_password_encrypted,omitempty"`
+       AuthToken              string `json:"-"`
+       AuthTokenEncrypted     string `json:"auth_token_encrypted,omitempty"`
+       UseAuthorizationHeader bool   `json:"use_authorization_header"`
+       AuthorizationHeader    string `json:"-"`
        // required: true
        UID int `json:"uid" binding:"Required"`
        // required: true
diff --git a/modules/repository/repo.go b/modules/repository/repo.go
index de6de3bda..1e69acb67 100644
--- a/modules/repository/repo.go
+++ b/modules/repository/repo.go
@@ -74,10 +74,11 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
        }
 
        if err = git.Clone(ctx, opts.CloneAddr, repoPath, git.CloneRepoOptions{
-               Mirror:        true,
-               Quiet:         true,
-               Timeout:       migrateTimeout,
-               SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+               Mirror:              true,
+               Quiet:               true,
+               Timeout:             migrateTimeout,
+               SkipTLSVerify:       setting.Migrations.SkipTLSVerify,
+               AuthorizationHeader: opts.AuthorizationHeader,
        }); err != nil {
                return repo, fmt.Errorf("Clone: %w", err)
        }
@@ -95,11 +96,12 @@ func MigrateRepositoryGitData(ctx context.Context, u *user_model.User,
                        }
 
                        if err := git.Clone(ctx, wikiRemotePath, wikiPath, git.CloneRepoOptions{
-                               Mirror:        true,
-                               Quiet:         true,
-                               Timeout:       migrateTimeout,
-                               Branch:        "master",
-                               SkipTLSVerify: setting.Migrations.SkipTLSVerify,
+                               Mirror:              true,
+                               Quiet:               true,
+                               Timeout:             migrateTimeout,
+                               Branch:              "master",
+                               SkipTLSVerify:       setting.Migrations.SkipTLSVerify,
+                               AuthorizationHeader: opts.AuthorizationHeader,
                        }); err != nil {
                                log.Warn("Clone wiki: %v", err)
                                if err := util.RemoveAll(wikiPath); err != nil {
diff --git a/services/migrations/gitea_uploader.go b/services/migrations/gitea_uploader.go
index 8a7533b3d..5de25dfb6 100644
--- a/services/migrations/gitea_uploader.go
+++ b/services/migrations/gitea_uploader.go
@@ -7,6 +7,7 @@ package migrations
 
 import (
        "context"
+       "encoding/base64"
        "fmt"
        "io"
        "os"
@@ -118,19 +119,24 @@ func (g *GiteaLocalUploader) CreateRepo(repo *base.Repository, opts base.Migrate
        r.DefaultBranch = repo.DefaultBranch
        r.Description = repo.Description
 
+       if opts.UseAuthorizationHeader {
+               opts.AuthorizationHeader = fmt.Sprintf("Authorization: Basic %s", base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", opts.AuthUsername, opts.AuthPassword))))
+       }
+
        r, err = repo_module.MigrateRepositoryGitData(g.ctx, owner, r, base.MigrateOptions{
-               RepoName:       g.repoName,
-               Description:    repo.Description,
-               OriginalURL:    repo.OriginalURL,
-               GitServiceType: opts.GitServiceType,
-               Mirror:         repo.IsMirror,
-               LFS:            opts.LFS,
-               LFSEndpoint:    opts.LFSEndpoint,
-               CloneAddr:      repo.CloneURL, // SECURITY: we will assume that this has already been checked
-               Private:        repo.IsPrivate,
-               Wiki:           opts.Wiki,
-               Releases:       opts.Releases, // if didn't get releases, then sync them from tags
-               MirrorInterval: opts.MirrorInterval,
+               RepoName:            g.repoName,
+               Description:         repo.Description,
+               OriginalURL:         repo.OriginalURL,
+               GitServiceType:      opts.GitServiceType,
+               Mirror:              repo.IsMirror,
+               LFS:                 opts.LFS,
+               LFSEndpoint:         opts.LFSEndpoint,
+               CloneAddr:           repo.CloneURL, // SECURITY: we will assume that this has already been checked
+               Private:             repo.IsPrivate,
+               Wiki:                opts.Wiki,
+               Releases:            opts.Releases, // if didn't get releases, then sync them from tags
+               MirrorInterval:      opts.MirrorInterval,
+               AuthorizationHeader: opts.AuthorizationHeader,
        }, NewMigrationHTTPTransport())
 
        g.sameApp = strings.HasPrefix(repo.OriginalURL, setting.AppURL)