Debugging tips#

See Emscripten’s page about debugging which has extensive info about the various debugging options available. The Wasm Binary Toolkit is super helpful for analyzing .wasm, .so, .a, and .o files.

Also whenever you can reproduce a bug in chromium make sure to use a chromium-based browser (e.g., chrome) for debugging. They are better at it.

Run prettier on pyodide.asm.js#

Before doing any debugger I strongly recommend running npx prettier -w pyodide.asm.js. This makes everything much easier.

Building with symbols or source maps#

Building with symbols retains the names of all internal functions, but will not prevent function calls from being inlined away and does not retain a source map to the original C code. Symbols do not cause much of a slowdown and only cause a minor increase in code size. Building with source maps disables many optimizations such as inlining and slows down execution by quite a lot.

To rebuild with symbols you can use the following:

touch src/core/jsproxy.h
PYODIDE_SYMBOLS=1 make -C cpython rebuild-all
PYODIDE_SYMBOLS=1 make all-but-packages

To use source maps, use PYODIDE_SOURCEMAPS=1 instead of PYODIDE_SYMBOLS=1. To get the benefits of source maps, you need the C/C++ devtoools support Chrome extension.

When source maps are present and the devtools extension is enabled, you will not be able to see the raw disassembly, which makes it hard to inspect the state of the program in the debugger because there is no support for executing C expressions. The extension provides no way to see the raw assembly without disabling the extension and refreshing the page.

Linker error: function signature mismatch#

You may get linker errors as follows:

wasm-ld: error: function signature mismatch: some_func
>>> defined as (i32, i32) -> i32 in some_static_lib.a(a.o)
>>> defined as (i32) -> i32 in b.o

This is especially common in Scipy. Oftentimes it isn’t too hard to figure out what is going wrong because it told you the both the symbol name (some_func) and the object files involved (this is much easier than the runtime version of this error!). If you can’t tell what is going on from looking at the source files, it’s time to pull out wasm-objdump. In this case a.o is part of some_static_lib.a so you first need to get it out with ar -x some_static_lib.a a.o. Now we can check if a.o imports or defines some_func. To check for imports, use wasm-objdump a.o -j Import -x | grep some_func. If a.o is importing some_func you should see a line like: - func[0] sig=1 <env.some_func> <- env.some_func in the output.

If not, you will see nothing or things like some_func2. To check if a.o defines some_func (this is a bit redundant because you can conclude whether or not does from whether it imports it) we can use: wasm-objdump a.o -j Function -x | grep some_func, if a.o defines some_func you will see something like: - func[0] sig=0 <some_func>.

Now the question is what these signatures mean (though we already know this from the linker error). To find out what signature 0 is, you can use wasm-objdump a.o -j Type -x | grep "type\[0\]".

Using this, we can verify that a.o imports some_func with signature (i32, i32) -> i32 but b.o exports it with signature (i32) -> i32, hence the linker error.

This process works in basically the same way for already-linked .so and .wasm files, which can help if you get the load-time version of this linker error.

Misencoded Wasm#

On a very rare occasion you may run into a misencoded object file. This can cause different tools to crash, wasm-ld may panic, etc. wasm-objdump will just generate a useless error message. In this case, I recommend wasm-objdump -s --debug 2>&1 | grep -i error -C 20 (or pipe to less), which will result in more diagnostic information. Sometimes the crash happens quite a lot later than the actual error, look for suspiciously large constants, these are often the first sign of something gone haywire.

After this, you can get out a hex editor and consult the WebAssembly binary specification Cross reference against the hex addresses appearing in wasm-objdump --debug. With enough diligence you can locate the problem.

Debugging RuntimeError: function signature mismatch#

First recompile with a source map as suggested in Building with symbols or source maps, and disable the C/C++ DevTools extension (if it’s enabled).

The browser console will show something like the following. Click on the innermost stack trace:

fpcast stack trace

Clicking the offset will (hopefully) take you to the corresponding wasm instruction, which should be a call_indirect. If the offset is too large (somewhere between 0x0200000 and 0x0300000) you will instead see ;; text is truncated due to size, see Dealing with ;; text is truncated due to size. In this example we see the following:

wasm bad call_indirect instruction

So we think we are calling a function pointer with signature (param i32 i32) (result i32) meaning that it takes two i32 inputs and returns one i32 output. Set a breakpoint by clicking on the address, then refresh the page and run the reproduction again. Sometimes these are on really hot code paths (as in the present example) so you probably only want to set the breakpoint once Pyodide is finished loading. If your reproduction passes through the breakpoint multiple times before crashing you can do the usual chore of counting how many times you have to press “Resume” before the crash. Suppose you’ve done all this, and we’ve got the vm stopped at the bad instruction just before crashing:

wasm bad function pointer

The bottom value on the stack is the function pointer. In this case it’s the fourth item on the stack, so you can type the following into the console:

> pyodide._module.wasmTable.get(stack[4].value) // stack[4].value === 13109
< ƒ $one() { [native code] }

So the bad function pointer’s symbol is one! Now clicking on $one brings you to the source for it:

function pointer signature

and we see the function pointer has signature (param $var0 i32) (result i32), meaning it takes one i32 input and returns one i32 output. Note that if the function had void return type it might look like (param $var0 i32 $var1 i32) (with no result). Confusion between i32 and void return type is the single most common cause of this error.

Now we basically know the cause of the trouble. You can look up cfunction_call in the CPython source code with the help of ripgrep and locate the line that generates this call, and look up one in the appropriate source and find the signature. Another approach to locate the call site would be to recompile with -g3 and use source maps Building with symbols or source maps to locate the problematic source code. With the same process of reproduce crash ==> click innermost stack frame ==> see source file and line where the error occurs. In this case we see that the crash is on the line:

result = _PyCFunction_TrampolineCall(meth, self, args);

in the file /src/cpython/build/Python-3.11.0dev0/Objects/methodobject.c. Unfortunately, source maps are useless for the harder problem of finding the callee because compiling with -g3 increases the number of function pointers so the function pointer we are calling is in a different spot. I know of no way to determine the bad function pointer when compiling with -g3.

Sometimes (particularly with Scipy/OpenBLAS/libf2c) the issue will be a mismatch between (param i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32) (result i32) and (param i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32) (result i32)

(14 vs 15 parameters) which might be a little hard to discern. I copy the signature into the Javascript console and run "i32 ... i32".split(" ").length in this case.

Dealing with ;; text is truncated due to size#

If you are debugging and run into the dreaded ;; text is truncated due to size error message, the solution is to compile a modified version of Chrome devtools with a larger wasm size cap. Surprisingly, this is not actually all that hard.

These instructions are adapted from here: https://www.diverto.hr/en/blog/2020-08-15-WebAssembly-limit/

In short,

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
./fetch devtools-frontend
cd devtools-frontend

Apply the following change:

--- a/front_end/entrypoints/wasmparser_worker/WasmParserWorker.ts
+++ b/front_end/entrypoints/wasmparser_worker/WasmParserWorker.ts
@@ -55,7 +55,7 @@ export function dissambleWASM(
     const lines = [];
     const offsets = [];
     const functionBodyOffsets = [];
-    const MAX_LINES = 1000 * 1000;
+    const MAX_LINES = 12 * 1000 * 1000;
     let chunkSize: number = 128 * 1024;
     let buffer: Uint8Array = new Uint8Array(chunkSize);
     let pendingSize = 0;

Then build with:

gn gen out/Default
autoninja -C out/Default

then

cd out/Default/resources/inspector_overlay/
python http.server <some_port>

and then you can start a version of chrome using the modified devtools:

chrome --custom-devtools-frontend=http://localhost:<some_port>/