pip downloads packages twice if the data-dist-info-metadata is set

[fiannac]

Description

I'm using a custom package index that implements PEP-658. When I install packages through that index specifying the --no-cache-dir flag pip downloads the package twice.

Expected behavior

No response

pip version

23.0.1

Python version

3.9.7

OS

CentOS 7.9

How to Reproduce

Run pip install --no-cache-dir --index-url PEP-658index -v package

Output

pip install --no-cache-dir --index-url http://localhost:8000/simple/ -v numba
Using pip 23.0.1 from ... (python 3.9)
Looking in indexes: http://localhost:8000/simple/
Collecting numba
  Obtaining dependency information for numba from http://localhost:8000/.../numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
  Downloading http://localhost:8000/.../numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.8 kB)
Requirement already satisfied: setuptools in /.../site-packages (from numba) (59.5.0)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /.../site-packages (from numba) (0.39.1)
Requirement already satisfied: numpy<1.24,>=1.18 in /.../site-packages (from numba) (1.23.5)
Downloading http://localhost:8000/.../numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 105.5 MB/s eta 0:00:00
Downloading http://localhost:8000/.../numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 120.8 MB/s eta 0:00:00
Installing collected packages: numba
Successfully installed numba-0.56.4

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

What are you setting in data-dist-info-metadata? If it is set pip downloads the content it points to for metadata. It looks like your attribute points to the same wheel, which should not be the case.

[pradyunsg]

Could you share the HTML file that triggered this, as well as the contents of the METADATA file served by the host?

The metadata file seems to be distinct: numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata

[fiannac]

What are you setting in data-dist-info-metadata? If it is set pip downloads the content it points to for metadata. It looks like your attribute points to the same wheel, which should not be the case.

From PEP658:

The repository SHOULD provide the hash of the Core Metadata file as the data-dist-info-metadata attribute’s value using the syntax hashname=hashvalue, where hashname is the lower cased name of the hash function used, and hashvalue is the hex encoded digest. The repository MAY use true as the attribute’s value if a hash is unavailable.

I am setting the values to true since I do not have access to the hashes. I also think pip is downloading and using the metadata correctly:
Downloading http://localhost:8000/.../numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.8 kB)

[fiannac]

Could you share the HTML file that triggered this, as well as the contents of the METADATA file served by the host?

The metadata file seems to be distinct: numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata

I'm currently just proxying pypi.org, adding data-dist-info-metadata to all the a tags pointing to a wheel dist and unwrapping the wheels on the fly when a client asks for metatadata. So the html is just the project page provided by pypi.org and the metadata is the METADATA file inside the wheel.

[fiannac]

Could you share the HTML file that triggered this, as well as the contents of the METADATA file served by the host?

The metadata file seems to be distinct: numba-0.56.4-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata

I've uploaded on this repo an html file that triggers this, you can reproduce that locally following the setps in the readme

[pradyunsg]

Thank you for that @fiannac! That's really useful.

[pradyunsg]

/cc @pfmoore, who is also looking at the PEP 658 + PEP 714 situation in #12078. I reckon this is something we should try to fix in the first release using the new key name.

[pfmoore]

I'll take a look, but I can't promise I'll have time to work out what the problem is here, much less create a fix. So PRs would be welcome.

Worst case scenario, if no-one comes up with a fix for this, we'll have PEP 658/714 support in 23.2 with this issue unfixed. Does anyone think that's a serious problem (serious enough to pull #12078 from 23.2)? My instinct is to say no, we go ahead with #12078, but a double download does sound like a rather serious performance regression, for something that was supposed to be a performance improvement 🙁

[pfmoore]

I've tried setting a breakpoint in pip._internal.network.download in Downloader.__call__, just before the HTTP response is read and written to the target file. The breakpoint is only triggered once, but the Downloading filename.whl (NN kB) message is printed twice before the breakpoint is hit. So I'm pretty sure that the issue here is simply one of the "Downloading" message being logged twice, rather than the file actually being downloaded twice.

Obviously, it's misleading and needs to be fixed, but I'm reasonably confident that there isn't a performance issue here.

Before I downgrade this from release blocker, can anyone demonstrate that my analysis above is wrong?

Note that #12078 has now been merged, and PyPI is serving (some) metadata files, so this issue can be reproduced using pip main and PyPI. The command line I used was pip install --dry-run --no-deps --no-cache piel -vvv (I used the piel package simply because it has had a very recent release which has metadata available on PyPI).

[fiannac]

I've tried setting a breakpoint in pip._internal.network.download in Downloader.__call__, just before the HTTP response is read and written to the target file. The breakpoint is only triggered once, but the Downloading filename.whl (NN kB) message is printed twice before the breakpoint is hit. So I'm pretty sure that the issue here is simply one of the "Downloading" message being logged twice, rather than the file actually being downloaded twice.

When I run the same command and check the server-side logs, it really looks like the file is being downloaded twice:

Client:
pip install --dry-run --no-deps --no-cache --force-reinstall --index-url http://localhost:8082/simple/ abcabc
Collecting abcabc
  Obtaining dependency information for abcabc from http://localhost:8082/simple/abcabc/abcabc-0.0.7-py3-none-any.whl.metadata
  Downloading http://localhost:8082/simple/abcabc/abcabc-0.0.7-py3-none-any.whl.metadata (92 bytes)
Downloading http://localhost:8082/simple/abcabc/abcabc-0.0.7-py3-none-any.whl (2.4 kB)
Downloading http://localhost:8082/simple/abcabc/abcabc-0.0.7-py3-none-any.whl (2.4 kB)

Server:
127.0.0.1 - - [28/Jun/2023 14:12:21] "GET /simple/abcabc/ HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2023 14:12:21] "GET /simple/abcabc/abcabc-0.0.7-py3-none-any.whl.metadata HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2023 14:12:21] "GET /simple/abcabc/abcabc-0.0.7-py3-none-any.whl HTTP/1.1" 200 -
127.0.0.1 - - [28/Jun/2023 14:12:21] "GET /simple/abcabc/abcabc-0.0.7-py3-none-any.whl HTTP/1.1" 200 -