Debugging a SQLite WASM Heap Fragmentation Bug in a VS Code Extension

While building AS Notes (A wikilink-based knowledge management VS Code extension - https://github.com/appsoftwareltd/as-notes) - I ran into a subtle and genuinely interesting bug involving WebAssembly linear memory.

TLDR: sql.js loads SQLite into a WASM module with a single linear memory buffer. After indexing 18,000+ files that memory is large and fragmented. Reconstructing a Database() object on that same heap fails with "memory access out of bounds". The fix is a build-time esbuild plugin that injects a resetCache() function into the sql.js bundle, allowing the WASM module to be fully discarded and reloaded from scratch on rebuild.

Background

AS Notes maintains a local SQLite index of your markdown knowledge base - pages, wikilinks, aliases, and tasks. The index is built and queried using sql.js, which compiles SQLite to WebAssembly. No native binaries, no external processes. It runs entirely inside the VS Code extension host.

On first initialisation, the extension scans your workspace, indexes every markdown file, and persists the result to disk as index.db. This works correctly regardless of knowledge base size.

The rebuild command - AS Notes: Rebuild Index - is intended to be a clean re-scan from scratch. It should produce exactly the same result as a fresh initialisation. In practice, it was crashing.

The Symptom

Running the rebuild command on a 18k .md file knowledge base produced one of two failure modes depending on which iteration of the fix was in place at the time:

• "memory access out of bounds" - a WebAssembly runtime trap
• A progress notification ("AS Notes: Rebuilding Index...") that would appear and never resolve

The out-of-memory errors were misleading. System memory was at 41% of 32 GB. This is not a system memory problem.

First initialisation, by contrast, worked perfectly every time ...

Why Initialisation Works and Rebuild Doesn't

This asymmetry is the important clue.

On first initialisation, the extension creates a brand new SQL.Database() instance loaded from a fresh WASM module. The WASM linear memory starts clean. Every allocation is sequential and predictable.

For rebuild, the extension reuses the existing IndexService and its already-loaded sql.js module - the same module that just finished indexing 18,500 files.

Here is the problem. WASM linear memory can only grow; it cannot shrink. During a full scan, SQLite performs millions of allocations inside the WASM heap: B-tree nodes, page cache buffers, prepared statement objects, query results. Over 18,500 files with approximately 280,000 wikilinks, this amounts to hundreds of thousands of malloc/free cycles within the WASM allocator. free() returns blocks to the allocator's free list, but the WASM high-water mark never moves. After a full scan, the heap has grown to roughly 80 MB in a heavily fragmented state - a large allocation arena full of holes.

When Database.close() is called followed by new Database() on the same WASM module, the new database object is constructed on that same fragmented heap. Smaller allocations succeed by fitting into the holes. Larger contiguous allocations eventually cannot be satisfied, and the WASM allocator either traps with "memory access out of bounds" or stalls indefinitely.

The failure at ~1,618 files (consistently reproducible) is not a coincidence - it's the point at which the allocator exhausts its ability to find sufficient contiguous space in the fragmented linear memory for SQLite's internal structures.

The Dead Ends

Several approaches were attempted before the root cause was properly understood.

resetSchema() (DROP TABLE / CREATE TABLE) - the original rebuild strategy. This invalidated sql.js's internal prepared statement cache. Subsequent operations silently failed or stalled. This is what caused the hanging progress notification.

clearAllData() (DELETE FROM all tables) - safer, but operating on the same database instance that was potentially loaded from a corrupt index.db written during a prior failed resetSchema() session. If the on-disk state was corrupt, this path hung too.

close() → delete index.db → initDatabase() - created a new Database() on the same fragmented WASM heap. This resolved the corrupt file problem but introduced the "memory access out of bounds" crash for large knowledge bases.

Caching the WASM module - calling initSqlJs() once and reusing the module across initDatabase() calls. Correct approach for avoiding multiple WASM heap allocations, but when paired with the close/reinit rebuild path, it still suffered from heap fragmentation.

The Fix: Build-Time WASM Cache Reset

The root cause is that the only way to get a clean WASM heap is to load a completely new WASM module. Database.close() frees SQLite's internal structures but does not reset the WASM allocator's state. There is no public API in sql.js to force this.

Internally, sql.js initSqlJs() caches the result of loading the WASM binary in a module-scoped variable, initSqlJsPromise. If this variable is set, subsequent calls to initSqlJs() return the cached module - which is exactly the designed behaviour for normal usage. The problem is there is no way from outside the module to clear that cache.

The solution is to inject one at build time.

The esbuild Plugin

const sqlJsCacheResetPlugin = {
  name: 'sql-js-cache-reset',
  setup(build) {
    build.onLoad({ filter: /sql-wasm\.js$/ }, async (args) => {
      let contents = await fs.promises.readFile(args.path, 'utf8');
      // Inject a resetCache() function into the closure before module.exports
      contents = contents.replace(
        'module.exports = initSqlJs;',
        'initSqlJs.resetCache = function() { initSqlJsPromise = undefined; };\nmodule.exports = initSqlJs;'
      );
      return { contents, loader: 'js' };
    });
  },
};

This runs during the esbuild bundling step. It intercepts the sql-wasm.js source, locates the module.exports = initSqlJs; line at the bottom of the CommonJS wrapper, and inserts the resetCache function immediately before it. Because the injection happens inside the __commonJS closure generated by esbuild, the injected function has direct access to the initSqlJsPromise variable - the same one that initSqlJs() itself closes over.

The exported initSqlJs function now has a resetCache property that, when called, sets initSqlJsPromise to undefined. The next call to initSqlJs() then loads the WASM binary afresh, allocating a brand new linear memory buffer from 0 bytes. The old module - with its 80 MB fragmented heap - becomes unreachable and eligible for garbage collection.

Usage in IndexService

resetSchema() was made async and now does the following:

1. Close the current database (this.db!.close())
2. Call (initSqlJs as any).resetCache() - clears the cached WASM module
3. Set this.sqlModule = null
4. await initSqlJs() - loads a fresh WASM binary with clean linear memory
5. new this.sqlModule.Database() - creates an empty database on the new heap
6. Run PRAGMA foreign_keys = ON and SCHEMA_SQL

The rebuild command calls await indexService!.resetSchema() before fullScan().

The result: every rebuild is now equivalent to a first initialisation. Fresh WASM module, clean linear memory, empty database, no fragmentation, no stale prepared statements, no corrupt state from prior failed sessions.

There is a brief period during resetSchema() where the old and new WASM modules coexist - approximately 160 MB peak. This is acceptable and transient. For all other operations (periodic stale scans, file save handlers, completions), the original WASM module is reused for the lifetime of the extension session. Only the explicit rebuild path triggers a fresh load.

Implications for Periodic Scans

The periodic stale scan (staleScan()) does not have this problem. It processes only changed or new files since the last scan - typically 0–10 files per tick. The WASM heap grows during the initial full scan and then remains approximately stable. Periodic scans add a small, bounded amount of allocation activity. There is no accumulation that would cause the fragmentation problem seen during rebuild.

The question of whether periodic VACUUM would be beneficial is valid but premature. The current failure mode was caused by WASM heap fragmentation on the extension side, not by SQLite page fragmentation inside the database file itself. Periodic VACUUM addresses the latter, which is not the observed problem.

A Note on Diagnostics

The debugging process was complicated by the fact that both failure modes - the hanging notification and the WASM trap - surfaced no useful error messages by default. The stall in particular produces nothing: the withProgress callback simply never returns.

As part of this work, a LogService was added to the extension. When enabled (via the as-notes.enableLogging setting or the AS_NOTES_DEBUG=1 environment variable), it writes timestamped log lines to .asnotes/logs/as-notes.log. If you are working with a large knowledge base and experiencing index issues, enabling this setting is the first diagnostic step.

Conclusion

This is the kind of bug that is easy to misread. "Out of memory" suggests system RAM. "Memory access out of bounds" suggests a code bug. Neither is true here (my system RAM had plenty of room). The issue is WASM linear memory fragmentation caused by the non-shrinkable nature of WASM heap allocations, surfacing only when a large database session is followed by an attempt to reinitialise on the same module.

The build-time injection approach is pragmatic rather than elegant. It relies on the internal structure of the sql.js CommonJS bundle, which could change between versions. But it is isolated, testable (confirmed present in the built output), and solves a problem that has no clean public API solution. The esbuild plugin is eight lines and the patch point is unambiguous.

If you're using AS Notes with a large knowledge base, this fix is in the current release. The rebuild command now works reliably, regardless of how many prior sessions the WASM module has survived.

Install AS Notes from the VS Code Marketplace | Source on GitHub