Web Design Blog

How to Store Unlimited* Data in the Browser with IndexedDB

How to Store Unlimited* Data in the Browser with IndexedDB – SitePointSkip to main contentFree JavaScript Book!Write powerful, clean and maintainable JavaScript.RRP $11.95 This article explains the fundamentals of storing data in the browser using the IndexedDB API, which offers a far greater capacity than other client-side mechanisms.
Storing web app data used to be an easy decision. There was no alternative other than sending it to the server, which updated a database. Today, there’s a range of options, and data can be stored on the client.
Why Store Data in the Browser?
It’s practical to store most user-generated data on the server, but there are exceptions:
device-specific settings such as UI options, light/dark mode, etc.
short-lived data, such as capturing a range of photographs before choosing one to upload
offline data for later synchronization, perhaps in areas with limited connectivity
progressive web apps (PWAs) which operate offline for practical or privacy reasons
caching assets for improved performance
Three primary browser APIs may be suitable:

Web Storage
Simple synchronous name-value pair storage during or beyond the current session. It’s practical for smaller, less vital data such as user interface preferences. Browsers permit 5MB of Web Storage per domain.

Cache API
Storage for HTTP request and response object pairs. The API is typically used by service workers to cache network responses, so a progressive web app can perform faster and work offline. Browsers vary, but Safari on iOS allocates 50MB.

IndexedDB
A client-side, NoSQL database which can store data, files, and blobs. Browsers vary, but at least 1GB should be available per domain, and it can reach up to 60% of the remaining disk space.

OK, I lied. IndexedDB doesn’t offer unlimited storage, but it’s far less limiting than the other options. It’s the only choice for larger client-side datasets.
IndexedDB Introduction
IndexedDB first appeared in browsers during 2011. The API became a W3C standard in January 2015, and was superseded by API 2.0 in January 2018. API 3.0 is in progress. As such, IndexedDB has good browser support and is available in standard scripts and Web Workers. Masochistic developers can even try it in IE10.

This article references the following database and IndexedDB terms:

database: the top-level store. Any number of IndexedDB databases can be created, although most apps will define one. Database access is restricted to pages within the same domain; even sub-domains are excluded. Example: you could create a notebook database for your note-taking application.

object store: a name/value store for related data items, conceptually similar to collections in MongoDB or tables in SQL databases. Your notebook database could have a note object store to hold records, each with an ID, title, body, date, and an array of tags.

key: a unique name used to reference every record (value) in an object store. It can be automatically generated or set to a value within the record. The ID is ideal to use as the note store’s key.

autoIncrement: a defined key can have its value auto-incremented every time a record is added to a store.

index: tells the database how to organize data in an object store. An index must be created to search using that data item as criteria. For example, note dates can be indexed in chronological order so it’s possible to locate notes during a specific period.

schema: the definition of object stores, keys, and indexes within the database.

version: a version number (integer) assigned to a schema so a database can be updated when necessary.

operation: a database activity such as creating, reading, updating, or deleting (CRUD) a record.

transaction: a wrapper around one or more operations which guarantees data integrity. The database will either run all operations in the transaction or none of them: it won’t run some and fail others.

cursor: a way to iterate over many records without having to load all into memory at once.

asynchronous execution: IndexedDB operations run asynchronously. When an operation is started, such as fetching all notes, that activity runs in the background and other JavaScript code continues to run. A function is called when the results are ready.

The examples below store note records — such as the following — in a note object store within a database named notebook:
{
id: 1,
title: “My first note”,
body: “A note about something”,
date: ,
tags: [“#first”, “#note”]
}

The IndexedDB API is a little dated and relies on events and callbacks. It doesn’t directly support ES6 syntactical loveliness such as Promises and async/await. Wrapper libraries such as idb are available, but this tutorial goes down to the metal.

I’m sure your code is perfect, but I make a lot of mistakes. Even the short snippets in this article were refactored many times and I trashed several IndexedDB databases along the way. Browser DevTools were invaluable.
All Chrome-based browsers offer an Application tab where you can examine the storage space, artificially limit the capacity, and wipe all data:

The IndexedDB entry in the Storage tree allows you to examine, update, and delete object stores, indexes, and individual record:

(Firefox has a similar panel named Storage.)
Alternatively, you can run your application in incognito mode so all data is deleted when you close the browser window.
Check for IndexedDB Support
window.indexedDB evaluates true when a browser supports IndexedDB:
if (‘indexedDB’ in window) {

}
else {
console.log(‘IndexedDB is not supported.’);
}

It’s rare to encounter a browser without IndexedDB support. An app could fall back to slower, server-based storage, but most will suggest the user upgrade their decade-old application!
Check Remaining Storage Space
The Promise-based StorageManager API provides an estimate of space remaining for the current domain:
(async () = > {

if (!navigator.storage) return;

const
required = 10,
estimate = await navigator.storage.estimate(),

available = Math.floor((estimate.quota – estimate.usage) / 1024 / 1024);

if (available >= required) {
console.log(‘Storage is available’);

}

})();

This API is not supported in IE or Safari (yet), so be wary when navigator.storage can’t returns a falsy value.
Free space approaching 1,000 megabytes is normally available unless the device’s drive is running low. Safari may prompt the user to agree to more, although PWAs are allocated 1GB regardless.
As usage limits are reached, an app could choose to:
remove older temporary data
ask the user to delete unnecessary records, or
transfer less-used information to the server (for truly unlimited storage!)
Open an IndexedDB Connection
An IndexedDB connection is initialized with indexedDB.open(). It is passed:
the name of the database, and
an optional version integer
const dbOpen = indexedDB.open(‘notebook’, 1);

This code can run in any initialization block or function, typically after you’ve checked for IndexedDB support.
When this database is first encountered, all object stores and indexes must be created. An onupgradeneeded event handler function gets the database connection object (dbOpen.result) and runs methods such as createObjectStore() as necessary:
dbOpen.onupgradeneeded = event = > {

console.log(`upgrading database from ${ event.oldVersion } to ${ event.newVersion }…`);

const db = dbOpen.result;

switch( event.oldVersion ) {

case 0: {
const note = db.createObjectStore(
‘note’,
{ keyPath: ‘id’, autoIncrement: true }
);

note.createIndex(‘dateIdx’, ‘date’, { unique: false });
note.createIndex(‘tagsIdx’, ‘tags’, { unique: false, multiEntry: true });
}

}

};

This example creates a new object store named note. An (optional) second argument states that the id value within each record can be used as the store’s key and it can be auto-incremented whenever a new record is added.
The createIndex() method defines two new indexes for the object store:
dateIdx on the date in each record
tagsIdx on the tags array in each record (a multiEntry index which expands individual array items into an index)
There’s a possibility we could have two notes with the same dates or tags, so unique is set to false.
Note: this switch statement seems a little strange and unnecessary, but it will become useful when upgrading the schema.
An onerror handler reports any database connectivity errors:
dbOpen.onerror = err = > {
console.error(`indexedDB error: ${ err.errorCode }`);
};

Finally, an onsuccess handler runs when the connection is established. The connection (dbOpen.result) is used for all further database operations so it can either be defined as a global variable or passed to other functions (such as main(), shown below):
dbOpen.onsuccess = () = > {

const db = dbOpen.result;

};

Create a Record in an Object Store
The following process is used to add records to the store:

Create a transaction object which defines a single object store (or array of object stores) and an access type of “readonly” (fetching data only — the default) or “readwrite” (updating data).

Use objectStore() to fetch an object store (within the scope of the transaction).

Run any number of add() (or put()) methods and submit data to the store:
const

writeTransaction = db.transaction(‘note’, ‘readwrite’),

note = writeTransaction.objectStore(‘note’),

insert = note.add({
title: ‘Note title’,
body: ‘My new note’,
date: new Date(),
tags: [ ‘#demo’, ‘#note’ ]
});

This code can be executed from any block or function which has access to the db object created when an IndexedDB database connection was established.
Error and success handler functions determine the outcome:
insert.onerror = () = > {
console.log(‘note insert failure:’, insert.error);
};

insert.onsuccess = () = > {

console.log(‘note insert success:’, insert.result);
};

If either function is not defined, it will bubble up to the transaction, then the database handers (that can be stopped with event.stopPropagation()).
When writing data, the transaction locks all object stores so no other processes can make an update. This will affect performance, so it may be practical to have a single process which batch updates many records.
Unlike other databases, IndexedDB transactions auto-commit when the function which started the process completes execution.
Update a Record in an Object Store
The add() method will fail when an attempt is made to insert a record with an existing key. put() will add a record or replace an existing one when a key is passed. The following code updates the note with the id of 1 (or inserts it if necessary):
const

updateTransaction = db.transaction(‘note’, ‘readwrite’),

note = updateTransaction.objectStore(‘note’),

update = note.put({
id: 1,
title: ‘New title’,
body: ‘My updated note’,
date: new Date(),
tags: [ ‘#updated’, ‘#note’ ]
});

Note: if the object store had no keyPath defined which referenced the id, both the add() and put() methods provide a second parameter to specify the key. For example:
update = note.put(
{
title: ‘New title’,
body: ‘My updated note’,
date: new Date(),
tags: [ ‘#updated’, ‘#note’ ]
},
1
);

Reading Records from an Object Store by Key
A single record can be retrieved by passing its key to the .get() method. The onsuccess handler receives the data or undefined when no match is found:
const

reqTransaction = db.transaction(‘note’, ‘readonly’),

note = reqTransaction.objectStore(‘note’),

request = note.get(1);

request.onsuccess = () = > {

console.log(‘note request:’, request.result);
};

request.onerror = () = > {
console.log(‘note failure:’, request.error);
};

The similar getAll() method returns an array matching records.
Both methods accept a KeyRange argument to refine the search further. For example, IDBKeyRange.bound(5, 10) returns all records with an id between 5 and 10 inclusive:
request = note.getAll( IDBKeyRange.bound(5, 10) );

Key range options include:
The lower, upper, and bound methods have an optional exclusive flag. For example:
IDBKeyRange.lowerBound(5, true): keys greater than 5 (but not 5 itself)
IDBKeyRange.bound(5, 10, true, false): keys greater than 5 (but not 5 itself) and less than or equal to 10
Other methods include:
Reading Records from an Object Store by Indexed Value
An index must be defined to search fields within a record. For example, to locate all notes taken during 2021, it’s necessary to search the dateIdx index:
const

indexTransaction = db.transaction(‘note’, ‘readonly’),

note = indexTransaction.objectStore(‘note’),

dateIdx = note.index(‘dateIdx’),

request = dateIdx.getAll(
IDBKeyRange.bound(
new Date(‘2021-01-01’), new Date(‘2022-01-01’)
)
);

request.onsuccess = () = > {
console.log(‘note request:’, request.result);
};

Reading Records from an Object Store Using Cursors
Reading a whole dataset into an array becomes impractical for larger databases; it could fill the available memory. Like some server-side data stores, IndexedDB offers cursors which can iterate through each record one at a time.
This example finds all records containing the “#note” tag in the indexed tags array. Rather than using .getAll(), it runs an .openCursor() method, which is passed a range and optional direction string (“next”, “nextunique”, “prev”, or “preunique”):
const

cursorTransaction = db.transaction(‘note’, ‘readonly’),

note = cursorTransaction.objectStore(‘note’),

tagsIdx = note.index(‘tagsIdx’),

request = tagsIdx.openCursor(‘#note’);

request.onsuccess = () = > {

const cursor = request.result;

if (cursor) {

console.log(cursor.key, cursor.value);
cursor.continue();

}

};

The onsuccess handler retrieves the result at the cursor location, processes it, and runs the .continue() method to advance to the next position in the dataset. An .advance(N) method could also be used to move forward by N records.
Optionally, the record at the current cursor position can be:
Deleting Records from an Object Store
As well as deleting the record at the current cursor point, the object store’s .delete() method can be passed a key value or KeyRange. For example:
const

deleteTransaction = db.transaction(‘note’, ‘readwrite’),

note = deleteTransaction.objectStore(‘note’),

remove = note.delete(5);

remove.onsuccess = () = > {
console.log(‘note deleted’);
};

A more drastic option is .clear(), which wipes every record from the object store.
Update a Database Schema
At some point it will become necessary to change the database schema — for example, to add an index, create a new object store, modify existing data, or even wipe everything and start again. IndexedDB offers built-in schema versioning to handle the updates — (a feature sadly lacking in other databases!).
An onupgradeneeded function was executed when version 1 of the notebook schema was defined:
const dbOpen = indexedDB.open(‘notebook’, 1);

dbOpen.onupgradeneeded = event = > {

console.log(`upgrading database from ${ event.oldVersion } to ${ event.newVersion }…`);

const db = dbOpen.result;

switch( event.oldVersion ) {

case 0: {
const note = db.createObjectStore(
‘note’,
{ keyPath: ‘id’, autoIncrement: true }
);

note.createIndex(‘dateIdx’, ‘date’, { unique: false });
note.createIndex(‘tagsIdx’, ‘tags’, { unique: false, multiEntry: true });
}

}

};

Presume another index was required for note titles. The indexedDB.open() version should change from 1 to 2:
const dbOpen = indexedDB.open(‘notebook’, 2);

The title index can be added in a new case 1 block in the onupgradeneeded handler switch():
dbOpen.onupgradeneeded = event = > {

console.log(`upgrading database from ${ event.oldVersion } to ${ event.newVersion }…`);

const db = dbOpen.result;

switch( event.oldVersion ) {

case 0: {
const note = db.createObjectStore(
‘note’,
{ keyPath: ‘id’, autoIncrement: true }
);

note.createIndex(‘dateIdx’, ‘date’, { unique: false });
note.createIndex(‘tagsIdx’, ‘tags’, { unique: false, multiEntry: true });
}

case 1: {
const note = dbOpen.transaction.objectStore(‘note’);
note.createIndex(‘titleIdx’, ‘title’, { unique: false });
}

}

};

Note the omission of the usual break at the end of each case block. When someone accesses the application for the first time, the case 0 block will run and it will then fall through to case 1 and all subsequent blocks. Anyone already on version 1 would run the updates starting at the case 1 block.
Index, object store, and database updating methods can be used as necessary:
All users will therefore be on the same database version … unless they have the app running in two or more tabs!
The browser can’t allow a user be running schema 1 in one tab and schema 2 in another. To resolve this, a the database connection onversionchange handler can prompt the user to reload the page:

db.onversionchange = () = > {

db.close();
alert(‘The IndexedDB database has been upgraded.nPlease reload the page…’);
location.reload();

};

Low Level IndexedDB
IndexedDB is one the of the more complex browser APIs, and you’ll miss using Promises and async/await. Unless your app’s requirements are simple, you’ll want roll your own IndexedDB abstraction layer or use a pre-built option such as idb.
Whatever option you choose, IndexedDB is one of the fastest browser data stores, and you’re unlikely to reach the limits of its capacity.
Related ArticlesUsing Redis with Node.jsJavaScriptBy