Tar ownership issues installing 1.19.1 in Docker

[eatonphil]

Hi! I am trying to install golangci-lint in a Go 1.12.10 debian image. When upgrading to 1.19.1 I am seeing errors trying to change ownership during install. Googling around suggests there is a tar command running without --no-same-owner.

I cannot replicate this locally because I must have more permissive Docker daemon privileges set up. But currently I'm unable to use 1.19.1 in a CI environment. 1.17.1 did not have this issue.

Step 1/3 : FROM golang:1.12.10
--
  | 1.12.10: Pulling from library/golang
  | 4a56a430b2ba: Pull complete
  | 4b5cacb629f5: Pull complete
  | 14408c8d4f9a: Pull complete
  | ea67eaa7dd42: Pull complete
  | a2a2197e145e: Pull complete
  | 33b8ad332cc6: Pull complete
  | aa30e10913db: Pull complete
  | Digest: sha256:61152d6f60bd8d83e6f652bb99544cd0dae96945318a355d32f4cafaa2a5dc39
  | Status: Downloaded newer image for golang:1.12.10
  | ---> f945ea07f224
  | Step 2/3 : RUN curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh\| sh -s -- -b $(go env GOPATH)/bin v1.19.1
  | ---> Running in f871522eb7af
  | golangci/golangci-lint info checking GitHub for tag 'v1.19.1'
  | golangci/golangci-lint info found version: 1.19.1 for v1.19.1/linux/amd64
  | tar: golangci-lint-1.19.1-linux-amd64/LICENSE: Cannot change ownership to uid 1678343834, gid 593637566: Invalid argument
  | tar: golangci-lint-1.19.1-linux-amd64/README.md: Cannot change ownership to uid 1678343834, gid 593637566: Invalid argument
  | tar: golangci-lint-1.19.1-linux-amd64/golangci-lint: Cannot change ownership to uid 1678343834, gid 593637566: Invalid argument
  | tar: Exiting with failure status due to previous errors
  | The command '/bin/sh -c curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh\| sh -s -- -b $(go env GOPATH)/bin v1.19.1' returned a non-zero code: 2
  | Unable to find image 'go1210:latest' locally
  | docker: Error response from daemon: pull access denied for go1210, repository does not exist or may require 'docker login'.
  | See 'docker run --help'.
  | Could not build manager binary
  | 🚨 Error: The command exited with status 1

[tpounds]

Hi @eatonphil thanks for the report! I tried reproducing locally with the following Dockerfile contents but was unable to trigger this issue.

FROM golang:1.12.10

RUN curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | \
    sh -s -- -b $(go env GOPATH)/bin v1.19.1

Can you provide any additional details about the CI environment you are using (docker version, CI system, etc.)? Having a bit more information may help us pinpoint the issue.

[lawrencegripper]

Hi, I'm seeing the same issue with this in Circle CI.

Here is a failing build..

https://circleci.com/gh/lawrencegripper/azbrowse/281

The build is:

https://github.com/lawrencegripper/azbrowse/blob/3491c0b9d0fa917686cfb91f5b146367d357629a/.circleci/config.yml#L1-L12

and the script it runs is

https://github.com/lawrencegripper/azbrowse/blob/3491c0b9d0fa917686cfb91f5b146367d357629a/scripts/install_ci_tools.sh#L4-L5

I'm going to play around and see if I can get this working, will post back what I find.

[lawrencegripper]

So using the SSH to failed agent I can run the following and get a failure

root@da91e551efe5:~# curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s v1.19.1
golangci/golangci-lint info checking GitHub for tag 'v1.19.1'
golangci/golangci-lint info found version: 1.19.1 for v1.19.1/linux/amd64
tar: golangci-lint-1.19.1-linux-amd64/LICENSE: Cannot change ownership to uid 1678343834, gid 593637566: Invalid argument
tar: golangci-lint-1.19.1-linux-amd64/README.md: Cannot change ownership to uid 1678343834, gid 593637566: Invalid argument
tar: golangci-lint-1.19.1-linux-amd64/golangci-lint: Cannot change ownership to uid 1678343834, gid 593637566: Invalid argument
tar: Exiting with failure status due to previous errors

But if I run v1.18.0 or v1.19.0 all is well

root@da91e551efe5:~# curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s v1.18.0
golangci/golangci-lint info checking GitHub for tag 'v1.18.0'
golangci/golangci-lint info found version: 1.18.0 for v1.18.0/linux/amd64
golangci/golangci-lint info installed ./bin/golangci-lint

------------

root@da91e551efe5:~# curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh| sh -s v1.19.0
golangci/golangci-lint info checking GitHub for tag 'v1.19.0'
golangci/golangci-lint info found version: 1.19.0 for v1.19.0/linux/amd64
golangci/golangci-lint info installed ./bin/golangci-lint

[lawrencegripper]

This seems like a good writeup of what's going on in a different context.

microsoft/vscode-remote-release#28 (comment)

Maybe the changes to how GoReleaser is run as part of v1.19.1 have caused the file ownership in the tars to be different.

c427c61#diff-b67911656ef5d18c4ae36cb6741b7965L63

Yeah looks like this may be the cause. Here is a comparison between the two tars in terms of the user who created them.

Still no clear why this affects CircleCI but doesn't occur locally when running in docker.

[lawrencegripper]

This explains why this occurs in CircleCI and not locally. TLDR:

CircleCI runs Docker containers with userns enabled in order to securely run customers’ containers. The host machine is configured with a valid UID/GID for remapping. This UID/GID must be in the range of 0 - 65535

https://circleci.com/docs/2.0/high-uid-error/#problem

[lawrencegripper]

Here is a quick workaround, using the following command you can add the required --no-same-owner into the install command.

curl -sfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sed 's/tar -/tar --no-same-owner -/g' | sh -s v1.19.1

I've added a PR to the godownloader repo to add this flag into the generated installer script.

[tpounds]

@eatonphil, @lawrencegripper I did a quick check against the last few release versions and it definitely seems like a change in the permissions in how the archives were generated.

$ tar vtf golangci-lint-1.17.1-linux-amd64.tar.gz
-rw-rw-r-- travis/travis 35148 2019-06-10 04:56 golangci-lint-1.17.1-linux-amd64/LICENSE
-rw-rw-r-- travis/travis 54000 2019-06-10 04:57 golangci-lint-1.17.1-linux-amd64/README.md
-rwxrwxr-x travis/travis 20808352 2019-06-10 05:07 golangci-lint-1.17.1-linux-amd64/golangci-lint

$ tar vtf golangci-lint-1.18.0-linux-amd64.tar.gz
-rw-r--r-- denis/staff   35148 2019-01-20 14:58 golangci-lint-1.18.0-linux-amd64/LICENSE
-rw-r--r-- denis/staff   57692 2019-09-10 05:05 golangci-lint-1.18.0-linux-amd64/README.md
-rwxr-xr-x denis/staff 20176896 2019-09-10 05:37 golangci-lint-1.18.0-linux-amd64/golangci-lint

$ tar vtf golangci-lint-1.19.0-linux-amd64.tar.gz
-rw-r--r-- denis/staff   35148 2019-01-20 14:58 golangci-lint-1.19.0-linux-amd64/LICENSE
-rw-r--r-- denis/staff   56557 2019-09-23 15:46 golangci-lint-1.19.0-linux-amd64/README.md
-rwxr-xr-x denis/staff 21110784 2019-09-23 16:40 golangci-lint-1.19.0-linux-amd64/golangci-lint

$ tar vtf golangci-lint-1.19.1-linux-amd64.tar.gz
-rw-r--r-- denis-isaev/LD\Domain Users 35148 2019-01-18 00:33 golangci-lint-1.19.1-linux-amd64/LICENSE
-rw-r--r-- denis-isaev/LD\Domain Users 56557 2019-09-24 05:12 golangci-lint-1.19.1-linux-amd64/README.md
-rwxr-xr-x denis-isaev/LD\Domain Users 21356544 2019-09-25 10:37 golangci-lint-1.19.1-linux-amd64/golangci-lint

I suspect the issue has something to do with how the latest tar.gz was generated (looks like a Windows machine). It had to be done manually since the goreleaser tools were broken when the release was tagged. I think there are a couple of options here.

1. Regenerate the release from a different machine/CI system
2. Wait until the v1.20.0 release

@jirfag Thoughts?

[tpounds]

I did one last extended check on the last two release tars.

$ tar xf golangci-lint-1.19.0-linux-amd64.tar.gz --to-command 'echo $TAR_FILENAME:$TAR_UID:$TAR_GID'                                                                          
golangci-lint-1.19.0-linux-amd64/LICENSE:501:50
golangci-lint-1.19.0-linux-amd64/README.md:501:50
golangci-lint-1.19.0-linux-amd64/golangci-lint:501:50

$ tar xf golangci-lint-1.19.1-linux-amd64.tar.gz --to-command 'echo $TAR_FILENAME:$TAR_UID:$TAR_GID'                                                                          
golangci-lint-1.19.1-linux-amd64/LICENSE:1678343834:593637566
golangci-lint-1.19.1-linux-amd64/README.md:1678343834:593637566
golangci-lint-1.19.1-linux-amd64/golangci-lint:1678343834:593637566

The most recent v1.19.1 release has extremely large uids/gids which is possibly overflowing the allowable range on some systems. There's a note about this in the Go stdlib and some related code to handle this. I haven't dug deeper to investigate exactly what is happening here.

see:
https://github.com/golang/go/blob/master/src/archive/tar/common.go#L25
golang/go@1420466

[lawrencegripper]

@tpounds outlined it above, the uid that is owner of the files is above 65535 in release 1.19.1 which is larger maximum allowed by CircleCI due to limitation imposed on docker via userns.

I did a PR to godownloader which updates their install script to use a flag which doesn't attempt to map the extracted file to the original owner uid and ensures the install script will work no matter what user created the tar... flag=--no-same-owner.

That's been merged so if you pull in that change and regen install.sh with godownloader all will be fixed. goreleaser/godownloader#136