21.3: Breaking editable installation with `No module named pip` error

[Panaetius]

Description

Our CI is failing due to version 21.3 release, in the pip wheel subprocess call during installation, with seemingly random packages being the culprit.

E.g.

  subprocess.CalledProcessError: Command '['/opt/hostedtoolcache/Python/3.7.12/x64/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpiq0bznxg', '--quiet', 'cryptography<3.5,>=3.4.1']' returned non-zero exit status 1.

Full Log attached below

The specific dependency it fails for seems completely random, in 5 subsequent runs it's been pathspec, cryptography, tabulate, rdflib and pyasn1.

I can reproduce the issue locally as well with Arch Linux, 21.3 fails, 21.2.4 works.
This only happens in editable mode (using -e flag), without it everything install fine.

Expected behavior

Renku Package to be installed without problem

pip version

21.3

Python version

3.7.8

OS

Arch Linux/Github Actions(Ubuntu)

How to Reproduce

1. clone https://github.com/SwissDataScienceCenter/renku-python
2. upgrade to pip 21.3
3. do pip install -e .[all]

Output

$ python -m pip install -e .[all]
/home/user/.pyenv/versions/renku-python/lib/python3.7/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/home/user/.pyenv/versions/renku-python/lib/python3.7/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
Obtaining file:///home/user/DEV/ETH/renku-python
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
  Build backend does not support editables, falling back to setup.py egg_info.
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /home/user/.pyenv/versions/renku-python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/user/DEV/ETH/renku-python/setup.py'"'"'; __file__='"'"'/home/user/DEV/ETH/renku-python/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-0jfl5x91
       cwd: /home/user/DEV/ETH/renku-python/
  Complete output (31 lines):
  /home/user/.pyenv/versions/renku-python/bin/python: No module named pip
  Traceback (most recent call last):
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 75, in fetch_build_egg
      subprocess.check_call(cmd)
    File "/home/user/.pyenv/versions/3.7.8/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/home/user/.pyenv/versions/renku-python/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpppr5o_gk', '--quiet', 'zc.relation<1.2,>=1.1']' returned non-zero exit status 1.
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/home/user/DEV/ETH/renku-python/setup.py", line 266, in <module>
      "Development Status :: 4 - Beta",
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 152, in setup
      _install_setup_requires(attrs)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 809, in fetch_build_eggs
      replace_conflicting=True,
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 768, in resolve
      replace_conflicting=replace_conflicting
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1051, in best_match
      return self.obtain(req, installer)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1063, in obtain
      return installer(requirement)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 877, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 77, in fetch_build_egg
      raise DistutilsError(str(e)) from e
  distutils.errors.DistutilsError: Command '['/home/user/.pyenv/versions/renku-python/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpppr5o_gk', '--quiet', 'zc.relation<1.2,>=1.1']' returned non-zero exit status 1.
  ----------------------------------------
WARNING: Discarding file:///home/user/DEV/ETH/renku-python. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[teije01]

Experiencing the same problems, also

/opt/conda/envs/vdsdb/bin/python3.7: No module named pip

Same with other python versions. Downgrading pip resolved our problem

[sbidoul]

Ok, this seems to be related to the PEP 660 implementation.

Since this project has a pyproject.toml without build-system table, it seems that pip<21.3 did first prepare the metadata using the setuptools.build_meta:__legacy__ build backend, then called setup.py develop, so doing a mix of PEP 517 and legacy installation.

Now pip first tries a PEP 660 editable metadata preparation and, since the build backend does not support PEP 660, it falls back to a legacy metadata preparation (setup.py egg_info), which should in theory be equivalent but, for some reason I don't fully understand, does not work for your project.

[sbidoul]

@Panaetius can you test if #10577 fixes the issue ?

You can install it with pip install -U "pip @ git+https://github.com/pypa/pip@refs/pull/10577/head"

[sbidoul]

Note I tentatively added this to a 21.3.1 bugfix milestone, since it is a regression, but this does not mean doing an actual bugfix release has been decided yet.

[Panaetius]

It gets a lot further in the install process but then unfortunately has the same error when trying to install things:

https://gist.github.com/Panaetius/5d9b29b3694b310e567bc803fb9878a8

Thank you for looking into this!