Version 0.17.0 Release Notes¶
Pyodide 0.17.0 is a major step forward from previous versions. It includes major maintenance improvements, a thorough redesign of the central APIs, and careful elimination of error leaks and memory leaks.
There have been a large number of backwards incompatible changes due to these design improvements. Our hope is that we have fixed a significant portion of the issues and that future releases will have fewer breaking changes.
New Features¶
Asyncio Support¶
We added full support for asyncio, including a new Python event loop that
schedules tasks on the browser event loop, support for top level await in
pyodide.runPythonAsync, and implementations of await for JsProxy
and PyProxy, so that it is possible to await a Python awaitable from
Javascript and a Javascript thenable from Python. This allows seamless
interoperability:
pyodide.runPython(`
async def test():
from js import fetch
# Fetch the Pyodide packages list
r = await fetch("packages.json")
data = await r.json()
# return all available packages
return data.dependencies.object_keys()
`);
let test = pyodide.globals.get("test");
// test returns a coroutine, we can await the coroutine
// from Javascript and it will schedule it on the Python event loop
result = await test();
console.log(result); // ["asciitree", "parso", "scikit-learn", ...]
Error Handling¶
Errors can now be thrown in Python and caught in Javascript or thrown in Javascript and caught in Python.
Support for this is integrated at the lowest level, so calls between Javascript and C functions behave as expected. The error conversion code is generated by C macros which makes implementing and debugging new logic dramatically simpler.
function jserror(){
throw new Error("ooops!");
}
pyodide.runPython(`
from js import jserror
from pyodide import JsException
try:
jserror()
except JsException as e:
print(str(args)) # prints "TypeError: ooops!"
`);
Python “builtin” Modules implemented in Javascript¶
It is now simple to add a Python module implemented in Javascript using
pyodide.registerJsModule:
let my_module = {
foo(x){
return x*x + 1;
},
bar(y, z){
return y*z + y + z;
}
};
pyodide.registerJsModule("my_mod", my_module);
pyodide.runPython(`
from my_mod import foo, bar
foo(7) # 50
bar(9, 5) # 59
`);
(Added in PR #1146)
New Conversion APIs¶
We added several new conversion APIs to give more explicit control over the foreign function interface. In particular, the goal was to make it easier for users to avoid leaking memory.
For the basic use cases, we have PyProxy.toJs and JsProxy.to_py
which respectively convert Python objects to Javascript objects and Javascript
objects to Python objects. We also added also “wrong-way” conversion functions
pyodide.to_js and pyodide.toPy which are particularly helpful for
when returning values across languages or to give extra control over the
behavior of called functions.
The promise handler methods JsProxy.then, JsProxy.catch, and
JsProxy.finally_ were particularly hard to use without leaking memory so
they have been updated with internal magic to automatically manage the memory.
For more advanced use cases where control over the life cycle of a PyProxy is
needed, there are create_proxy and create_once_callable.
API Changes¶
We removed as_nested_list and deprecated pyimport. The old loading method
using languagePluginURL and languagePluginLoader is also deprecated, use
instead globalThis.loadPyodide. Access to Python globals via
pyodide.globals has also changed: pyodide.globals.x ==>
pyodide.globals.get("x") (pyodide.globals.x is still supported but is
deprecated).
Changes to type translations¶
In the past we found that one of the major pain points in using Pyodide occurred when an object makes a round trip from Python to Javascript and back to Python and comes back different. This violates the expectations of the user and forces inelegant workarounds (see #780 and #892 among others).
The type conversion module has significantly reworked in v0.17 with the goal
that round trip conversions of objects between Python and Javascript produces an
identical object. That is, Python -> JS -> Python conversion produces an object
that’s now equal to the original object, and JS -> Python -> JS conversion
verifies the === equality operation.
We also made extensive additions to the type conversions test suite and documentation, though gaps remain.
See issue #900 for some of the discussion.
(Mostly implemented in PRs #1152 and #1167, see also #1186 which )
Changes to buffer translations¶
The buffer translation code in previous versions was less flexible, leaked memory, and had serious bugs including use after free (#749) and buffer overflow errors.
We completely reworked these: buffers are now proxied like most other objects.
In simple use cases they can be converted with a copy using PyProxy.toJs
and JsProxy.to_py. We added new APIs PyProxy.getBuffer,
JsProxy.assign, and JsProxy.assign_to which give more fine-grained
control, though they are not yet as ergonomic as they could be.
Maintenance and Bug fixes¶
Pyodide version 0.17.0 comes with an enormous amount of maintenance work. There is still a lot of work to do be done before Pyodide will be ready for a version 1.0, but we made very significant headway. Significant improvements were made to every component of Pyodide, including the build system, the test suite and continuous integration, and the core Pyodide code.
Upstream Emscripten¶
We finally completed the migration to the latest version of Emscripten (https://emscripten.org/) compiler toolchain which uses the upstream LLVM backend. This allows us to take advantage of recent improvements to the toolchain reducing package size and execution time.
For instance, the scipy package shrank dramatically from 92 MB to 15 MB. Scipy is now automatically cached in browsers, greatly improving the usability of scientific Python packages that depend on scipy, such as scikit-image and scikit-learn. The size of the base Pyodide environment with only the CPython standard library shrank from 8.1 MB to 6.4 MB.
On the performance side, the latest toolchain comes with a 25% to 30% run time improvement for pure Python code.
Because we are now using a very recent Emscripten version, we were able to upstream many of our patches to the compiler toolchain.
Core C Code maintenance¶
The core C code base was improved with new macros that streamline the most common tasks and help with consistency, and in addition to the extensive additions to the test suite we performed a manual audit of the entire C codebase to locate leaks and logic errors.
We fixed many of the long standing bugs caused by inconsistencies in the
behavior of JsProxy. There used to be two different code paths for
producing a JsProxy and two different code paths for calling a
JsProxy leading to four different behaviors, all of which were presented
errors in some cases. This fixed numerous bugs, including issues #461,
#768, #788, and #1123. The number of surprises you can
expect when using the foreign function interface has gone way down.
Similarly, we consolidated the entrypoints and removed redundant APIs. Now every
public entrypoint into C code has been consolidated into the file pyproxy.js.
The number of places where C calls into Javascript is much more diverse, but
these call sites have all been wrapped in a special macro that automatically
converts Javascript functions to use the CPython calling convention. They can
even be passed as function pointers to C functions from the Python standard
library!
Fatal error detection¶
Because we have wrappers on all the entrypoints to C code and all of the exitpoints, we can detect fatal errors: during normal execution we have careful control over how the stack unwinds and if an exception comes out of an unexpected place, we report a fatal error. This leads to faster, better bug reports and less confusion (using Pyodide after a fatal error occurs can lead to very strange behavior).
Fixed error leaks and memory leaks¶
Errors in C code must be manually returned to the calling code, and memory must be manually released. We refactored all of the existing C code to apply a consistent approach to memory management and error handling, based on the try / finally idiom. As much as possible, references are only freed in the finally block at the end of the function to make it easier to check correctness.
This allowed us to fix a large number of error leaks and memory leaks. We now have pretty complete test coverage for memory leaks. The error coverage is less complete, though we added fault injection tests for every entrypoint.
(See for instance #1340)