Testing and benchmarking

Testing

Requirements

Install the following dependencies into the default Python installation:

pip install pytest selenium pytest-instafail pytest-httpserver

Install geckodriver and chromedriver and check that they are in your PATH.

Running the test suite

To run the pytest suite of tests, type on the command line:

pytest src/ pyodide_build/ packages/*/test_*

There are 3 test locations,

  • src/tests/: general Pyodide tests and tests running the CPython test suite

  • pyodide_build/tests/: tests related to Pyodide build system (do not require selenium to run)

  • packages/*/test_*: package specific tests.

Manual interactive testing

To run manual interactive tests, a docker environment and a webserver will be used.

  1. Bind port 8000 for testing. To automatically bind port 8000 of the docker environment and the host system, run: ./run_docker

  2. Now, this can be used to test the Pyodide builds running within the docker environment using external browser programs on the host system. To do this, run: ./bin/pyodide serve

  3. This serves the build directory of the Pyodide project on port 8000.

    • To serve a different directory, use the --build_dir argument followed by the path of the directory.

    • To serve on a different port, use the --port argument followed by the desired port number. Make sure that the port passed in --port argument is same as the one defined as DOCKER_PORT in the run_docker script.

  4. Once the webserver is running, simple interactive testing can be run by visiting this URL: http://localhost:8000/console.html

Benchmarking

To run common benchmarks to understand Pyodide’s performance, begin by installing the same prerequisites as for testing. Then run:

make benchmark

Linting

Python is linted with flake8. C and Javascript are linted with clang-format.

To lint the code, run:

make lint

Testing framework

run_in_pyodide

Many tests simply involve running a chunk of code in Pyodide and ensuring it doesn’t error. In this case, one can use the run_in_pyodide decorate from pyodide_build/testing.py, e.g.

from pyodide_build.testing import run_in_pyodide

@run_in_pyodide
def test_add():
    assert 1 + 1 == 2

In this case, the body of the function will automatically be run in Pyodide. The decorator can also be called with arguments. It has two configuration options — standalone and packages.

Setting standalone = True starts a standalone browser session to run the test (the session is shared between tests by default). This is useful for testing things like package loading.

The packages option lists packages to load before running the test. For example,

from pyodide_build.testing import run_in_pyodide

@run_in_pyodide(standalone = True, packages = ["regex"])
def test_regex():
    import regex
    assert regex.search("o", "foo").end() == 2