IndexedDB is Weird

Programming

Why? Well:

The IndexedDB API is callback-based.

With JavaScript being single-threaded, a blocking API would mean fully blocking the page, render and basic user interaction included, while the request is being processed. Although this is apparently good-enough for JSON.parse(), the W3C decided to make the IndexedDB API non-blocking.

The first drafts for IndexedDB are from 2009, with the first implementations becoming available in 2011. In contrast, async only reached the main browsers in late 2016. Without async, a JavaScript API that avoids blocking needs to be callback-based. So, even though JavaScript got itself an out from callback hell in the form of promises and async functions, IndexedDB did not. Of course, you can alleviate the pain by making your own wrappers, but that means extra code and extra bugs.

/** @return {Promise<IDBDatabase>} */
function openDB() {
    return new Promise((resolve, reject) => {
        const request = indexedDB.open("test");
        request.onupgradeneeded = () => request.result.createObjectStore("items");
        request.onsuccess = () => resolve(request.result);
        request.onerror = () => reject(request.error);
    });
}

The IndexedDB API is very limited.

While even NoSQL databases usually provide a flexible DSL to query the database, IndexedDB gives you few tools to work with. For instance, to get data from the database, you can do the equivalent of SELECT * FROM items WHERE … :

And that’s it. Everything else, you need to do yourself.

No JOIN. Sure, NoSQL database generally don’t have them. But the thing is that the initial plan was to just expose an SQLite database to JavaScript. However, Mozilla disagreed, arguing that a Web standard should not depend on a specific database. Mozilla did a number of great things, but this was not one. In any case, the result is that IndexedDB lets you talk to a full-featured SQL database with tons of useful features, but you have to do it in a convoluted way that exposes virtually none of them.

No LIMIT. Combined with the callback-based design, that means that, to do SELECT * FROM items ORDER BY FIELD DESC LIMIT ?count, you need to do something like this:

/**
 * @param {number} count
 * @return {Promise<Item[]>}
 */
function getLastItems(count) {
    return new Promise((resolve, reject) => {
        const transaction = db.transaction("items");
        const objectStore = transaction.objectStore("items");
        const index = objectStore.index("field");
        const request = index.openCursor(null, "prev");
        /** @type{Item[]} */
        const items = [];
        request.onsuccess = () => {
            const result = request.result;
            if (result) {
                items.push(result.value);
            }
            if (result && items.length < count) {
                result.continue();
            } else {
                items.reverse();
                resolve(items);
            }
        };
        request.onerror = reject;
    });
}

No AND, no OR, etc. Basically, all your logic becomes a mess of many JavaScript method calls. More JavaScript, just what we needed!

IndexedDB transactions autocommit

In many languages, and database transaction automatically commits when the object that holds it goes out of scope. IndexedDB did something similar. But the devil is in the details. From MDN’s introduction to IndexedDB:

Transactions are tied very closely to the event loop. If you make a transaction and return to the event loop without using it then the transaction will become inactive. The only way to keep the transaction active is to make a request on it.

In short: you cannot hold a transaction over an await point. That means that you cannot do anything atomatically that would require some async handling. Oh, and you cannot opt out of this behavior.

I have been hit by this while implementing a progress bar do show progress while hundreds of thousands of items were being added in a transaction. Let’s start with a naive version.

let processed = 0;
const toProcess = 100000;
function updateProgress() {
    processed += 1;
    progressBar.style.width = `${processed / total * 100}%`;
}
for (let i = 0; i < toProcess; i++) {
    const request = objectStore.put({name: "item" + i}, i);
    request.onsuccess = updateProcess;
}

This kinds of work: the whole thing freeze for a few seconds, then the progress bar actually moves in a reasonable way. The freezing is due do the 100,000 calls to object.Store.put() themselves taking a lot of time. Once the loop above is done, the function returns control to the event loop, which handles the requests and triggers the onsuccess callbacks progressively.

To make the progress bar smoother, we need to also update it in the for-loop.

const transaction = db.transaction(["items"], "readwrite");
const total = 2 * items.length;
let processed = 0;
function updateProgress() {
    processed += 1;
    progressBar.style.width = `${processed / total * 100}%`;
}
const objectStore = transaction.objectStore("items");
for (const item of items) {
    const request = objectStore.put(item);
    request.onsuccess = updateProgress;
    processed += 1;
    if (processed % 1000 === 0) {
        progressBar.style.width = `${processed / total * 100}%`;
        await sleep(100);
    }
}
transaction.oncomplete = () => {
    progressBar.style.width = "100%";
    button.classList.remove("spinning");
};

Since running JavaScript blocks the rendering of the page, we need to return control to the event loop so that the update of the progress bar gets rendered. Wait a second! Returning control to the event loop?

A request was placed against a transaction which is currently not active, or which is finished.

Ah, yes, of course. The transaction gets committed the first time we try to update the progress bar. Then, we get an error on the following loop iteration.

Let’s see how we fix this. Yes, there is a proper way to do that.

IndexedDB lies about transaction lifetimes

Trying to figure out how to work around this, ChatGPT suggested running a loop in a background that would run dummy requests to keep the transaction alive. Spoiler: this was a terrible idea, which is quite expected from any LLM whenever you do something slightly out of the mainstream. Skipping on all the alternatives I tried, the fundamental idea is to call keepTransactionAlive(objectStore) right after the transaction is opened, with:

async function keepTransactionAlive(objectStore) {
    while (true) {
        console.log(".");
        await putItem(objectStore, {}, 0);
    }
}

This function keeps dispatching new requests for a dummy object. But, however much I made sure that keepTransactionAlive() was keeping outstanding requests on the transaction, it did not keep the other requests from failing:

Nevertheless! If you are paying attention, you should notice the number in the blue pill at the bottom. This tells us that keepTransactionAlive() is still happily chugging along, generating tens of thousands of requests, which are all succeeding. In addition, it is easy to confirm that the transaction is never closed:

No items were actually committed to the database. In other words, keepTransactionAlive() is actually working, preventing the transaction from committing. But it also means that the description of transaction lifetimes is misleading. I should have expected it; in the same paragraph as the quote from before:

Transaction lifetimes are really very simple but it might take a little time to get used to.

When technical documentation insists on something being “really very simple”, it usually means that it is anything but simple.

The Solution

As far as I can tell, what actually happens is that, once you return to the event loop, the transaction rejects any new requests, except those performed in a callback. Once you know this rule, the solution is easy to figure out.

const transaction = db.transaction(["items"], "readwrite");
const objectStore = transaction.objectStore("items");
const toProcess = items.length;
let processed = 0;
nonBlockingPuts(objectStore, items, () => {
    processed += 1;
    progressBar.style.width = `${5 + (processed / toProcess) * 95}%`;
});

Where nonBlockingPuts() is defined as:

/** Dispatch many PUT requests without blocking the main thread
 *
 * @param {IDBObjectStore} objectStore
 * @param {any[]} puts
 * @param {() => void} callback
 */
function nonBlockingPuts(objectStore, puts, callback) {
    let pendingRequests = 0;
    function dispatchRequest() {
        while (pendingRequests < 100 && puts.length !== 0) {
            const request = objectStore.put(puts.pop());
            request.onsuccess = () => {
                pendingRequests -= 1;
                dispatchRequest();
                callback();
            };
            pendingRequests += 1;
        }
    }
    dispatchRequest();
}

This function will gradually dispatch a number of puts requests, while ensuring that new requests are done from the onsuccess callback. You cannot easily map this to async code, but it works!