Pushing large amount of data via git-lfs results in HTTP 413 after some time

[simonszu]

• Gitea version (or commit ref): 1.2.3
• Git version: 2.11.0 on server, 2.15.0 on client. git-lfs version 2.3.4 on client and server
• Operating system: Debian Stable
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant

The try-instance seems to use git-lfs (if enabled) via https and does not accept my login credentials

• Log gist:

Description

I have problems pushing a very large repository (1.8GB) with git lfs to my gitea instance. git pushes the first 80MB, after that, the counter of already uploaded data jumps between weird values (e.g. it increases, decreases, increases again, and so on), and after some time, i get lots of LFS client errors, which are HTTP 413 status codes reported back from gitea. The last few lines say that there is an authorization error for info/lfs/objects/batch and the push process stops.
This is reproducible for several push attempts. I tried setting the log level to Debug, but the log file stays quite empty.

So i'm asking for help for how to set up Debug logging (i cannot find a list of proper log level options), and i want to notice for this issue in general.

I have configured a Bitbucket Repository as a second remote, and it accepts the big push without problems, so i think it is a server related issue.

[lunny]

This maybe a reverse proxy upload size setting problem?

[simonszu]

I am not sure. I have played around with different upload size limitations in my reverse proxy, and finally disabled it completely (client_max_body_size 0;). The result now is this:

$ git push -u tharan master                                                                                                                                                [09:08:21]
Git LFS: (42 of 467 files, 396 skipped) 148.06 MB / 471.06 MB, 84.70 MB skipped                                                                                                                                                               Git LFS: (42 of 467 files, 396 skipped) 151.75 MB / 471.06 MB, 84.70 MB skipped                                                                                                                                                               Git LFS: (42 of 467 files, 396 skipped) 151.75 MB / 471.06 MB, 84.70 MB skipped                                                                                                                                                               
Git LFS: (42 of 467 files, 396 skipped) 151.75 MB / 471.06 MB, 84.70 MB skipped         

The first of the 4 Git LFS lines was the output of the push command, which stalled after some time. The other 3 lines are the result of repeatedly pressing the Enter key. I entered the command on 09:08, now it's 09:54, and i am seeing no progress.

[simonszu]

Please note that i have another project with LFS data which sums up to about 10MB. This is working perfectly fine with gitea.

[rokups]

I am experiencing same issue. I have over 10GB repository and pushing it over ssh fails exactly the same way. I managed to push about 1GB leaving process run over night restarting git push command when it fails. Oddly enough i changed remote to use https and now it has pushed over 800MB without a single error. Could the issue be in ssh?

P.S. @simonszu you should use email as login credentials for https login as for some reason it does not accept usernames (on my local instance at least). looks like it accepts both login and email, but asks it 3 times. weird.

[rokups]

I copied my git repository into docker container and tried to push it to gitea without any proxies in the middle. This what happens:

Git LFS: (0 of 1779 files, 104427 skipped) 0 B / 2.92 GB, 10.00 GB skipped                                                                                                                                                                                        
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/9742a4f0128be197d24483be6d7cf6567ad9f512c59b718ce43833041e81d899: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/2b31a74d70bf26b78b19937a58629b2c18457c5ad13b4d842e2bfba406e36b63: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/1b741218078d119793c28a20bd633c1f8cd1bd9ce15c8e55aef6f96677c9e554: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/5cb116a997aa62cdfe317135f8812d0c506ee96e57a5136d51e2b2fb7918b0f4: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/9164d2aaa1dab2236a35887d62d75ccca70e8b371a02974c18827816e93ba0f8: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/b8f50aa3eb233ec647ca31e96c0fbe9fa3fbd4ad5303cffc329d33545eea6244: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/13d10c8b3b0ba6bf05151ad205f97e5bb7d489f31f409878b388ef96705620d9: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/071aa9e3774e320c64d65e48d4dbe2ca00c614beaa4a42ee7ee9a719fd6f8d2e: dial tcp 172.21.0.2:443: getsockopt: connection refused                                                                      
LFS: Put https://privatehostname.com/user/repo.git/info/lfs/objects/dac70040a45900e5f9df2e66cb4d15f6a61f2c1bc4597e412c45f1aecc1ec64b: dial tcp 172.21.0.2:443: getsockopt: connection refused

Curious thing is that remote is set to http://localhost:3000/user/repo.git.

[liu-kan]

I added this in nginx configure file

http {
     # ..........
     # at the END of this segment!
     client_max_body_size 3000m;
}

But, I still got

Git LFS: (0 of 2 files, 3 skipped) 0 B / 3.96 GB, 287.45 MB skipped                                                                                                                                               
LFS: Client error: https://git...d.git/info/lfs/objects/6fb4175dcacea6d2ba94e3e82de5b9c322e4aa662812dc3c9448e6212847524f from HTTP 413
LFS: Client error: https://git....d.git/info/lfs/objects/f3fe319831272c3feeb55198c3d1e1b3a165081f1290a20bd11a27f33128e326 from HTTP 413
error: failed to push some refs to 'ssh://git@git....d.git'

I used gitea-docker version ca30698

[sapk]

I can replicate similar bug with sqlite. I got server side error
RemoveLFSMetaObjectByOid: database table is locked.
I will have a look at it but it seems not related to size but more about concurrency.

[dpopiashvili]

Hi

Is there any resolution on this? I'm trying to push a test repository containing ~400MB of LFS files and it fails with authorization error.

c:\temp\Test>git push -u origin master
Git LFS: (0 of 2 files) 0 B / 472.87 MB
batch response: Repository or object not found: https://dato0011:19861967@try.gitea.io/Dato0011/test2.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
error: failed to push some refs to 'https://dato0011@try.gitea.io/Dato0011/test2.git'

Was it fixed by #4035 ?

[simonszu]

I need to wait for the next release. I like bleeding edge, but compiling from HEAD isn't always a great idea. ;)

[singinwhale]

I tried running the master branch and it fixed a similar problem for me: My issue looked like this:

Gitea Bootlog:

2018/06/08 14:09:46 [I] Git Version: 2.11.0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    │
2018/06/08 14:09:46 [I] SQLite3 Supported                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      │
2018/06/08 14:09:46 [I] Run Mode: Production                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   │
2018/06/08 14:09:46 [I] Listen: unix:///tmp/gitea.socket                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       │
2018/06/08 14:09:46 [I] LFS server enabled

Nginx reverse Proxy log:

[08/Jun/2018:14:17:01 +0200] "PUT /singinwhale/MoeWars.git/info/lfs/objects/fdbabf7c652dc5122b2e552189342d5b67393b89eb27086b5bce7471d0bd9815 HTTP/1.1" 401 12 "-" "git-lfs/2.4.2 (GitHub; windows amd64; go 1.8.3; git 6f4b2e98)"
[08/Jun/2018:14:17:01 +0200] "POST /singinwhale/MoeWars.git/info/lfs/objects/batch HTTP/1.1" 401 26 "-" "git-lfs/2.4.2 (GitHub; windows amd64; go 1.8.3; git 6f4b2e98)"

Git output during upload:

# git push singinwhale.com
Uploading LFS objects:   0% (0/1), 34 MB | 537 KB/s

Git output after upload:

Uploading LFS objects:   0% (0/1), 0 B | 544 KB/s, done
batch response: Authentication required: Authorization error: https://git.singinwhale.com/singinwhale/MoeWars.git/info/lfs/objects/batch
Check that you have proper access to the repository
error: failed to push some refs to 'git@git.singinwhale.com:singinwhale/MoeWars.git'

I am pushing a commit with a tag (1.0.0) to an existing repo on gitea via SSH and LFS. I have set the new LFS_HTTP_AUTH_EXPIRY_MINUTES value to 3220 (~2 days) as introduced by #4035 and now the 178MB file uploads just fine. As opposed to the Error 401 I got previously. OP had an Error 413 though so I dont know if it is exactly the same issue.

[lafriks]

I think #4035 fixes this issue, reopen if issue still persists