Maintainer information#

Making a release#

For branch organization we use a variation of the GitHub Flow with the latest release branch named stable (due to ReadTheDocs constraints).

Preparation for making a major release#

Generally we make a tracking issue with a title like “0.25.0 release planning”.

Follow the steps in Updating packages.

Read the changelog and tidy it up by adding subsections and proof reading it.

Generate the list of contributors for the release at the end of the changelog entry with

git shortlog -s LAST_TAG.. | cut -f2- | grep -v '\[bot\]' | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//' | fold -s

where LAST_TAG is the tag for the last release.

Make a pull request with these changes titled “Rearrange changelog for 0.25.0 release” and merge it.

Preparation for making a minor release#

Make a branch called backports-for-v.vv.v:

git checkout stable
git pull upstream
git checkout -b backports-for-0.23.1

Locate the commits you want to backport in the main branch and cherry pick them:

git cherry-pick <commit-hash>

Make a pull request from backports-for-0.23.1 targeting the stable branch. If you’re using the github cli this can be done with:

gh pr create -w -B stable

In the pull request description add a task:

- [ ] Merge don't squash

This pull request is a good place to @mention various people to ask if they have opinions about what should be backported.

Add an extra commit organizing the changelog into sections and editing changelog messages. Generate the list of contributors for the release at the end of the changelog entry with

git shortlog -s LAST_TAG.. | cut -f2- | grep -v '\[bot\]' | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//' | fold -s

where LAST_TAG is the tag for the last release. Make a branch from main called changelog-for-v.vv.v and apply the same changelog rearrangements there.

Merge changelog-for-v.vv.v and backports-for-v.vv.v and then follow the relevant steps from Release Instructions.

Preparation for making an alpha release#

Name the first alpha release x.x.xa1 and in subsequent alphas increment the final number. No prepration is necessary. Don’t update anything in the changelog. Follow the relevant steps from Release Instructions.

Release Instructions#

  1. From the root directory of the repository run

    ./tools/bump_version.py --new-version <new_version>
# ./tools/bump_version.py --new_version <new_version> --dry-run

    and check that the diff is correct with git diff. Try using ripgrep to make sure there are no extra old versions lying around e.g., rg -F "0.18", rg -F dev0, rg -F dev.0.

  2. (Skip for alpha release.) Add a heading to the changelog indicating version and release date

  3. Make a PR with the updates from steps 1 and 2. Merge the PR.

  4. (Major release only.) Rename the stable branch to a release branch for the previous major version. For instance if last release was, 0.20.0, the corresponding release branch would be 0.20.X:

    git fetch upstream stable:stable
git branch 0.20.X stable
git push -u upstream 0.20.X

  5. Create a tag X.Y.Z (without leading v) and push it to upstream,

    git checkout main
git pull upstream
git tag X.Y.Z
git push upstream X.Y.Z

    Wait for the CI to pass and create the release on GitHub.

  6. (Major release only). Create a new stable branch from this tag,

    git checkout main
git checkout -B stable
git push upstream stable --force

  7. (Major or alpha but not minor release.) Set the version number back to the development version. If you just released 0.22.0, set the version to 0.23.0.dev0. If you just released 0.22.0a1 then you’ll set the version to 0.22.0.dev0. Make a new commit from this and push it to upstream.

    git checkout main
./tools/bump_version.py --new-version 0.23.0.dev0
git add -u
git commit -m "0.23.0.dev0"
git push upstream main

Fixing documentation for a released version#

Cherry pick the corresponding documentation commits to the stable branch. Use git commit --amend to add [skip ci] to the commit message.

Updating the Docker image#

Anyone with an account on hub.docker.com can follow the following steps:

  1. Make whatever changes are needed to the Dockerfile.

  2. Build the docker image with docker build . in the Pyodide root directory. If the build succeeds, docker will give you a hash for the built image.

  3. Use python ./tools/docker_image_tag.py to find out what the new image tag should be. Tag the image with:

    docker image tag <image-hash> <your-docker-username>/pyodide-env:<image-tag>

  4. Push the image with:

    docker image push <your-docker-username>/pyodide-env:<image-tag>

  5. Replace the image in .circleci/config.yml with your newly created image. Open a pull request with your changes to Dockerfile and .circleci/config.yml.

  6. When the tests pass and the pull request is approved, a maintainer must copy the new image into the pyodide dockerhub account.

  7. Then replace the image tag in .circleci/config.yml, .devcontainer/devcontainer.json, and run_docker with the new image under the pyodide dockerhub account.

It’s also possible to update the docker image by pushing your changes to the Dockerfile to a branch in the pyodide/pyodide repo (not on a fork) and clicking Run workflow on https://github.com/pyodide/pyodide/actions/workflows/docker_image.yml.

Updating packages#

Before updating the Python version and before making a major Pyodide release, we try to update all packages that are not too much trouble. Run

make -C packages update-all

to update all packages and make a pull request with these changes. There will be build/test failures, revert the packages that fail the build or tests and make a note to update them independently.

Upgrading pyodide to a new version of CPython#

Prerequisites#

The desired version of CPython must be available at:

  1. The specific release section of https://www.python.org/downloads

  2. https://hub.docker.com/_/python

  3. https://github.com/actions/python-versions/releases

If doing a major version update, save time by Updating packages first.

Steps#

  1. Follow the steps in “Updating the Docker image” to create a docker image for the new Python version.

  2. Make sure you are in a Python virtual environment with the new version of Python and with requirements.txt installed. (It is also possible to work in the docker image as an alternative.)

  3. Update the Python version in Makefile.envs

  4. Update the Python version in the following locations:

    • .github/workflows/main.yml

    • docs/conf.py

    • docs/development/contributing.md

    • docs/development/building-and-testing-packages.md

    • environment.yml

    • .pre-commit-config.yaml

    • pyodide-build/pyodide_build/tools/pyo3_config.ini (two places)

    • pyproject.toml

    (TODO: make this list shorter.)

  5. Rebase the patches:

    • Clone cpython and cd into it. Checkout the Python version you are upgrading from. For instance, if the old version is 3.11.3, use git checkout v3.11.3 (Python tags have a leading v.) Run

      git am ~/path/to/pyodide/cpython/patches/*

    • Rebase the patches onto the new version of Python. For instance if updating from Python v3.11.3 to Python 3.12.1:

      git rebase v3.11.3 --onto v3.12.1

    • Resolve conflicts / drop patches that have been upstreamed. If you have conflicts, make sure you are using diff3:

      git config --global merge.conflictstyle diff3

    • Generate the new patches:

      rm ~/path/to/pyodide/cpython/patches/*
git format-patch v3.12.1 -o ~/path/to/pyodide/cpython/patches/

  6. Try to build Python with make -C cpython. Fix any build errors. If you modify the Python source in tree after a failed build it may be useful to run make rebuild.

  7. Try to finish the build with a top level make. Fix compile errors in src/core and any link errors. It may be useful to apply upgrade_pythoncapi.py --no-compat to the C extension in src/code. https://github.com/python/pythoncapi-compat/blob/main/upgrade_pythoncapi.py

    The file most tightly coupled to the CPython version is src/core/stack_switching/pystate.c. Consult the following greenlet file to figure out how to fix it: https://github.com/python-greenlet/greenlet/blob/master/src/greenlet/TPythonState.cpp

  8. In the virtual environment with the new Python version, run

    python src/tests/make_test_list.py

    Then run the core tests pytest src/tests/test_core_python.py and either fix the failures or update src/tests/python_tests.yaml to skip or xfail them.

  9. Try to build packages with:

    pyodide build-recipes '*'

    Disable packages until the build succeeds. Then fix the build failures. In many cases, this just requires updating to the most recent version of the package. If you have trouble, try searching on the package’s issue tracker for “python 3.12” (or whatever the new version is). It’s best to create separate PRs for tricky package upgrades.

  10. Fix failing package tests.

Old major Python upgrades#

version

pr

3.12

#4435

3.11

#3252

3.10

#2225

3.9

#1637

3.8

#712

3.7

#77