Building and testing Python packages out of tree

Building and testing Python packages out of tree#

This is some information about how to build and test Python packages against Pyodide out of tree (for instance in your package’s CI or for use with private packages).

Pyodide currently only supports Linux for out of tree builds, though there is a good change it will work in MacOS too. If you are using Windows, try Windows Subsystem for Linux.

Building binary packages for Pyodide#

If your package is a pure Python package (i.e., if the wheel ends in py3-none-any.whl) then follow the official PyPA documentation on building wheels Otherwise, the procedure is as follows.

Install pyodide-build#

pip install pyodide-build

Set up Emscripten#

You need to download the Emscripten developer toolkit:

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk

then you can install the appropriate version of Emscripten:

PYODIDE_EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version)
./emsdk install ${PYODIDE_EMSCRIPTEN_VERSION}
./emsdk activate ${PYODIDE_EMSCRIPTEN_VERSION}
source emsdk_env.sh

If you restart your shell, you will need to run source emsdk_env.sh again.

Build the WASM/Emscripten wheel#

Change directory into the package folder where the setup.py or pyproject.toml file is located. You should be in a shell session where you ran source emsdk_env.sh. Then run

pyodide build

in the package folder . This command produces a wheel in the dist/ folder, similarly to the PyPA build command.

If you need to add custom compiler / linker flags to the compiler invocations, you can set the CFLAGS, CXXFLAGS and LDFLAGS environment variables. For instance, to make a debug build, you can use: CFLAGS=-g2 LDFLAGS=g2 pyodide build.

pyodide build invokes a slightly modified version of the pypa/build build frontend so the behavior should be similar to what happens if you do:

pip install build
python -m build

If you run into problems, make sure that building a native wheel with pypa/build works. If it does, then please open an issue about it.

Serve the wheel#

Serve the wheel via a file server e.g., python3.10 -m http.server --directory dist. Then you can install it with pyodide.loadPackage or micropip.install by URL.

Notes#

the resulting package wheels have a file name of the form *-cp310-cp310-emscripten_3_1_27_wasm32.whl and are compatible only for a given Python and Emscripten versions. In the Pyodide distribution, Python and Emscripten are updated simultaneously.
for now, PyPi does not support emscripten/wasm32 wheels so you will not be able to upload them there.

Testing packages against Pyodide#

Pyodide provides an experimental command line runner for testing packages against Pyodide. Using it requires nodejs version 14 or newer.

The way it works is simple: you can create a virtual environment with:

pyodide venv .venv-pyodide

Activate it just like a normal virtual environment:

source .venv-pyodide/bin/activate

As a warning, things are pretty weird inside of the Pyodide virtual environment because python points to the Pyodide Python runtime. Any program that uses Python and is sensitive to the current virtual environment will probably break.

You can install whatever dependencies you need with pip. For a pure Python package, the following will work:

pip install -e .

For a binary package, you will need to build a wheel with pyodide build and then point pip directly to the built wheel. For now, editable installs won’t work with binary packages.

# Build the binary package
pyodide build
# Install it
pip install dist/the_wheel-cp310-cp310-emscripten_3_1_20_wasm32.whl[tests]

To test, you can generally run the same script as you would usually do. For many packages this will be:

python -m pytest

but for instance numpy uses a file called runtests.py; the following works:

python runtests.py

and you can pass options to it just like normal. Currently subprocess doesn’t work, so if you have a test runner that uses subprocess then it cannot be used.

Build Github actions example#

Here is a complete example of a Github Actions workflow for building a Python wheel out of tree:

runs-on: ubuntu-22.04
  steps:
  - uses: actions/checkout@v4
  - uses: actions/setup-python@v5
    with:
       python-version: 3.11.2
  - run: |
      pip install pyodide-build>=0.23.0
      echo EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version) >> $GITHUB_ENV
  - uses: mymindstorm/setup-emsdk@v14
    with:
       version: ${{ env.EMSCRIPTEN_VERSION }}
  - run: pyodide build

For an example “in the wild” of a github action to build and test a wheel against Pyodide, see the numpy CI