- Management of Python-related tools and container images
- Table of Content (ToC)
- Overview
- Projects this repository helps to manage
- Procedures
Table of contents generated with markdown-toc
- ML Helpers GitHub repository
- Docker land page for
infrahelpers/python-light - Quay land page for
artificialintelligence/python-light - Distributions:
- Python 3.12:
- Python 3.11:
- Python 3.10:
- Python 3.9:
- Python 3.8:
- Native Python on latest Alpine:
- Badges:
- GitHub repsitory - Images for Data Processing Pipelines (DPP)
- Docker land page for
infrahelpers/dpp - Image tags:
jdk17-python3.12,jdk11-python3.12,jdk8-python3.12,jdk17-python3.11,jdk11-python3.11,jdk8-python3.11
- GitHub repsitory - Docker for C++ development
- Docker land page for
infrahelpers/cpppython - Quay land page for
cpppythondevelopment/base - Distributions:
ubuntu2404,ubuntu2204,ubuntu2004,debian12,debian11,centos9,centos8,centos7,fedora40,fedora39 - Badges:
- GitHub repsitory - Docker for Python (Jupyter, Pandas)
- Docker land page for
infrahelpers/python-jupyter - Quay land page for
artificialintelligence/python-jupyter - Distributions:
ubuntu2404,ubuntu2204,ubuntu2004,debian12,debian11,centos9,centos8 - Badges:
-
Image tags:
pyspark-emr6,pyspark-emr6-light,pyspark-emr-dbs,pyspark-emr-jdk11,py312-bookworm,py311-bookworm,py310-bookworm,py39-bookworm,py38-bookworm -
Badges:
-
Git repository: https://github.com/cpp-projects-showcase/docker-images
-
Add the commands to install the new Python version in the Dockerfile files of all the distriibutions. For instance, adding Python 3.12.0 to Ubuntu 22.04
$ cat ubuntu2204/Dockerfile
...
# Python 3.12.3
RUN pyenv install 3.12.3 && \
pyenv global 3.12.3 && \
python -mpip install -U pip pipenv
RUN pyenv global system || echo "No default system version of Python. Sticking to 3.12.3"
...- Keeping at least one of the older versions of Python (3.10.13 and 3.11.8 here) gives the teams managing downstream Docker images the time to catch up.
The ML Python Jupyter Docker images are built on top of the generic C++/Python images, described in the section above. The Python versions are therefore controlled by those Docker images.
The Python virtual environment specifications need however to be upgraded. Files to update:
- Git repository: https://github.com/machine-learning-helpers/docker-python-light
The default Python versions for light images (e.g., Alpine, Debian and Debian Slim), as maintained by Docker itself (on GitHub and on Docker Hub), are usually the latest Python stable versions, only a few days after they have been released. There is hence no need to upgrade the Python version on the light Docker images.
Reference: ML-related Python induction project
Note that this Git repository is also a sub-module of the Python Jupyter Docker image project
There are two folders in that project where the Python setup has to be updated:
- Root folder
ml_induction_pythonfolderPipfilerequirements.txt- Note that the
requirements.txtfile may be upgraded with:
- Note that the
$ pipenv install
pipenv requirements > requirements.txt
git add requirements.txt
git commit -m "Upgreded the Python dependencies"Then, the ML Docker project (see above) should be updated to point to the new Python ML induction head.
- Clone the ML Docker project and initialize sub-modules (if needed):
$ mkdir -p ~/dev/ml && git clone https://github.com/machine-learning-helpers/docker-python-jupyter ~/dev/ml/docker-python-jupyter
$ cd ~/dev/ml/docker-python-jupyter
$ git submodule init
$ git submodule update --recursive --force- Pull (with Git) the head of the
masterbranch on the ML induction module:
$ cd ~/dev/ml/docker-python-jupyter/notebook/induction
$ git checkout master
$ git pull- Add, commit and push (with Git) the newly updated module:
$ cd ~/dev/ml/docker-python-jupyter
$ git add notebook/induction
$ git commit -m "[Modules] Updated the Python ML induction module"
$ git pushReferences:
Files to update:
References:
Files to update:
Pipfilerequirements.txt- Copy that
requirements.txtfile asci-scripts/requirements.txtin OpenTravelData (OPTD)
- Copy that
References:
Files to update:
PipfilePipfile.lock- (potentially)
setup.py - (potentially)
pyproject.toml requirements.txtrequirements-dev.txt- (potentially)
.travis.yml - (potentially)
tox.ini
Publish the component on PyPi:
user@laptop$ cd ~/dev/geo/python-opentraveldata
user@laptop$ rm -rf dist
user@laptop$ python -mbuild
user@laptop$ PYPIURL="https://test.pypi.org"
user@laptop$ twine upload -u __token__ --repository-url ${PYPIURL}/legacy/ dist/*
user@laptop$ PYPIURL="https://pypi.org"
user@laptop$ #keyring set ${PYPIURL}/ __token__
user@laptop$ twine upload -u __token__ --non-interactive dist/*The result should available on PyPi, e.g., https://pypi.org/project/opentraveldata/0.0.9.post2/
References:
Files to update:
Publish the component on PyPi:
user@laptop$ rm -rf dist && mkdir dist
user@laptop$ python setup.py sdist bdist_wheel
user@laptop$ PYPIURL="https://test.pypi.org"
user@laptop$ twine upload -u __token__ --repository-url ${PYPIURL}/legacy/ dist/*
user@laptop$ PYPIURL="https://pypi.org"
user@laptop$ #keyring set ${PYPIURL}/ __token__
user@laptop$ twine upload -u __token__ --non-interactive dist/*The resulting artifact should be available on PyPi, e.g.: https://pypi.org/project/OpenTrepWrapper/0.7.7.post3/
- In a folder governed by
Pipfile:- To install the dedicated Python virtual environment:
$ pipenv install; pipenv install --dev- To update the dedicated Python virtual environment:
$ pipenv update- On MacOS, with the
psycopg2module, there may be some trouble at installation time related to an issue with SSL libraries. It can usually be solved with:
$ LDFLAGS="-I/usr/local/opt/openssl/include" CPPFLAGS="-L/usr/local/opt/openssl/lib" pipenv install psycopg2
Installing psycopg2
Adding psycopg2 to Pipfile's [packages]
✔ Installation Succeeded
$ pipenv install; pipenv install --dev