4.0.8 sdist tarball signature on pypi is bad

[dvzrv]

Hi! When trying to package 4.0.8 for Arch Linux I ran into a bad signature for the sdist tarball on pypi.org:

gpg --verify gitdb-4.0.8.tar.gz.asc
gpg: assuming signed data in 'gitdb-4.0.8.tar.gz'
gpg: Signature made 2021-10-23T02:39:33 CEST
gpg:                using RSA key EAF9CCFCD0876408F297C60A9CB5EE7895E8268B
gpg: BAD signature from "Sebastian Thiel (YubiKey USB-C) <byronimo@gmail.com>" [unknown]

To reproduce:

curl -L -O https://files.pythonhosted.org/packages/source/g/gitdb/gitdb-4.0.8.tar.gz -O https://files.pythonhosted.org/packages/source/g/gitdb/gitdb-4.0.8.tar.gz.asc && gpg --verify gitdb-4.0.8.tar.gz.asc

[Byron]

Thanks for reporting, I am able to reproduce the issue as well.

Now I wonder what could have happened there, since it looks 'like always' on my end. My gpg is able to properly sign commits, --show-signature considers these valid, so I hope this was just a cosmic ray.

[Byron]

curl -L -O https://files.pyAthonhosted.org/packages/source/g/gitdb/gitdb-4.0.9.tar.gz -O https://files.pythonhosted.org/packages/source/g/gitdb/gitdb-4.0.9.tar.gz.asc && gpg --verify gitdb-4.0.9.tar.gz.asc still says it's a bad signature.

I have no idea how to fix this or what the problem even is :/.

[dvzrv]

Can you create a signature locally to test whether it is also bad?

e.g. gpg --detach-sign gitdb-4.0.9.tar.gz and then verify it via gpg --verify gitdb-4.0.9.tar.gz.asc

How are you creating the signature and how is it pushed to pypi.org?

[Byron]

Great idea, as this easily reproduces the issue while showing what's going wrong:

gpg -u 27C50E7F590947D7273A741E85194C08421980C9 --detach-sign gitdb-4.0.9.tar.gz
gpg: error reading symlink '/proc/curproc/file': No such file or directory

It still creates a signature though, which might be a signature over no bytes at all. Initially I thought it would be related to me using gpg from nix-pkg, but now I am using the homebrew one with the same outcome.

Funnily enough, I am not the only one: https://stackoverflow.com/questions/69699986/gpg-error-reading-symlink-proc-curproc-file-no-such-file-or-directory . The issue seems to be all but harmless, and is probably related to breakage in MacOS 12/12.1. The latter one I am using.

By the looks of it there won't be valid signatures from me for a while until this is fixed. Now that I can reproduce it, I can keep trying as updates come in.

How are you creating the signature and how is it pushed to pypi.org?

	twine upload -s -i 27C50E7F590947D7273A741E85194C08421980C9 dist/*

Twine does it, and I wouldn't be surprised if it just calls gpg under the hood.

[dvzrv]

@Byron can you use a different machine/ a VM for an intermediate release?

[Byron]

In the linked stackoverflow question one of the answers says it's fixed in 2.3.3.1, so I upgraded to this one. Unfortunately, despite not showing the error message anymore, the signature is still bad right after creation. Maybe they also considered the issue harmless and squelched it. Maybe it's something very else.

Setting up a VM to sign should be possible and I will see when I get to it. Alternatively I think I could try MacGPG as I'd expect them to maintain a working version.

[Byron]

No, MacGPG and its binary have the same issue despite claiming support for MacOS 12. I'd think my key got corrupted, but that's something it would probably detect, besides, signing commits works and yields valid signatures.
So maybe gpg for detached signatures is thoroughly broken on MacOS right now (as are a few other things, like the linker 😅)

[dvzrv]

You could also have a look at sq.

[Byron]

I took a look and alongside the --sign-with=<path-to-gpg-wrapper> one should be able to make twine use sq for signing.

Now one would just have to bring in the GPG keys to sq. Probably all doable and desirable to open a new chapter on the GPG story which definitely left a bitter taste thus far.

[dvzrv]

Probably all doable and desirable to open a new chapter on the GPG story which definitely left a bitter taste thus far.

I agree. We also just recently switched the Arch Linux distribution keyring handling over to sq, as its tooling and CLI is less overengineered and nicer to use.

[dvzrv]

Any update on this? I unfortunately can not upgrade the package at the moment.

[Byron]

Sorry for the wait and the hassle it must be.

For now I don't know when I will spend the time to deal with this. It's certainly not helped by my expectation that it will be painful to touch a combination of python tooling and handle loose GPG keys.

At the latest it will happen once GitPython NG lands, the one that uses Gitoxide under the hood.

In the mean time, after each update of MacOS or GPG I try again to see if something changes, but have no hopes.

[dvzrv]

I am sorry, but I will now drop all PGP signature validation from all gitpython related packages (this ticket has been open since October 2021).

Having PGP signature validation does not justify the breakage that is caused by trying to make this work (see e.g. https://bugs.archlinux.org/task/73193) and I have spent considerable amounts of time trying to make this work by providing numerous tickets (and I do not see myself doing this anymore).
This is not feasible and as this has been broken for months already I would suggest you consider either properly fixing this or just dropping it. As it stands it is just not possible to rely on the PGP signatures for the gitpython projects :-/

[Byron]

Thanks for speaking up and sharing your experience, and I am sorry to be the cause of this hassle and hope that dropping PGP signatures will fix the issue.

Now that I know it's an option, I will do that right away which hopefully allows GitPython packages to become available again on arch linux. PGP was always a hassle for me as well and even though sequoia could be the solution, I am just unable to invest the time needed to migrate to it. It's sad that suddenly perfectly valid keys just fail to produce valid signatures in this particular setting, without error message or anything like that.