Reflections from Local-First Conf

I've been interested in Local-first software for a couple of years. Last week, I attended the world's first Local-First Conf. In this blog post, I will share some of my thoughts and reflections from the conference, as well as ideas about the role IPFS can play in this movement.

The conference, which was the first of its kind had a distinct energy and felt like the cusp of a new era of software. Software that enshrines more user-agency by design. If you haven't heard about it, I recommend reading the essay.

"In this article we propose “local-first software”: a set of principles for software that enables both collaboration and ownership for users. Local-first ideals include the ability to work offline and collaborate across multiple devices, while also improving the security, privacy, long-term preservation, and user control of data."

With ~150 attendees, it was small and homey. Everyone I spoke to was pretty experienced as a developer and seemed to share the same inkling that local-first software is going to be an important movement over the next decade.

There was a diverse mix of talks that loosely fall into the following categories:

The problem/solution space: philosophy of local-first and how it fits into the broader space, where we came from and where we're going, sync protocols, UCANs, and "barefoot developers".
Dev tools: (automerge, y.js, jazz, fireproof, DXOS, PowerSync, ElectricSQL, zero)
Techniques and experience reports from devs building local-first apps: (overtone, Linear, goodnotes, Anytype, and affine, Pitch's offline approach)
The commercial case for local-first: why Linear was built, and the unreasonable advantage of building local-first.

Overall, it was a really fun and inspiring conference and I look forward to engaging, learning, and building more — all while embracing the local-first principles.

Local-first architectures offer increased developer productivity

One theme that was shared by all the talks is how complex web applications have become, and how some of the traditional 3-tier approaches to building web applications, don't play well with local-first principles. An implied corollary of this insight is that for local-first software to become mainstream, we need better building blocks, i.e. CRDTs, and architectures to make it easier to build local-first software.

In the case of Linear (which is more offline-first than it is local-first), it's clear that they have invested significant resources into first getting their offline-first architecture right, and are now reaping the development productivity rewards, which Tuomas confidently attested to in terms of their ability to ship new features. For example, if I remember correctly, embracing local-first has allowed them to avoid both async hell in their frontend code, as well as all the additional state and logic around async state (loading and error states, and optimistic update logic). They achieved this by moving a lot of the async state-syncing logic into a worker thread.

SQL on the Web with WASM

It seems as if the natural divergence of solutions that comes with a "new" paradigm is now reaching a turning point of convergence, whereby many competing solutions are arriving at the same conclusions. At the very least, there was a sense that patterns and approaches are converging.

For example, SQL and relational databases are taking a new life in the browser. While typically thought of as a foundational piece of the 3-tier technology stack, relational databases are now powering local-first web apps as the embedded database that is packaged into the web app, whose state is then synced across devices.

Embedded databases are not a new concept. That's the very idea of SQLite. However, running SQLite in the browser is a pretty new development, thanks to WASM, with which you can compile compiled languages into a binary format that can run in the browser.

File systems on the Web

Where things get complicated with WASM is i/o, i.e., filesystem and network access. Historically, interaction with the File System in the browser was made possible with the introduction of the File System API, however, for security reasons, it's built in a way that isn't sufficiently performant and suitable (in-place reads/writes) for SQLite.

Thanks to the new Origin private file system API (OPFS), which extends the File System API, it's now possible to get a synchronous and efficient filesystem API exclusively inside a Web Worker (when synchronous). Since OPFS isn't visible to the user (and is origin-isolated) it circumvents many of the security guardrails that come with the standard File System API.

WASM and OPFS make it possible for SQLite to run in the browser. To my amazement, it's now also possible to run a forked version of PostgreSQL called PGlight with WASM, thanks to the work of Sam Willis.

Having a local SQL database is just one part of the puzzle. The magic that makes real-time collaboration possible is when it is paired with a synching protocol or solution, that replicates and syncs databases across devices and users. Some of the solutions include ElectricSQL and PowerSync.

CRDTs

Amongst the other approaches for reconciling the state of different users are conflict-free replicated data types commonly known as CRDTs.

CRDTs employ algorithms that allow multiple users to make changes in a way that deterministically converges to the same end state, i.e. eventually consistent. CRDTs do not require coordination which allows concurrent changes while offline making them a good fit for local-first.

One of the takeaways from the conference and my research on CRDTs is that they've reached a level of maturity feasible for production use cases. For example, Automerge, one of the popular CRDT implementations, was rewritten in Rust as part of the Automerge 2.0 and is an order of magnitude faster than earlier versions.

I'm not an expert on CRDTs, so if you're interested in learning more, check out localfirstweb.dev which has many learning resources.

I still have some open questions like how to handle permissions, synchronization across devices, and best practices, which I hope to explore more.

Where does IPFS fit into Local-first?

I've been active in the IPFS ecosystem for two years and think there's a good overlap in principles and values between IPFS and local-first.

IPFS at its core is about content-addressing. Most systems that embrace content-addressing allow local writes that can be synched layer, e.g. Git is a classic example of this.

The IPFS Principles document gives a broad definition of IPFS implementation requirements, that essentially says: "as long as you address and expose operations using CIDs and verify bytes match the CID" you are an IPFS implementation.

I generally think this principle is great for experimentation and improvement, but it comes at the cost of fragmentation, ambiguity, and confusion in the IPFS ecosystem. For example, is libp2p part of IPFS? What about IPLD and multiformats? The Network Indexer, which is in the defaults of Kubo? IPFS Gateways? IPNS and DNSLink?

If you pose this question to folks in the community, I'm pretty sure you will get a wide spectrum of answers.

The Fission team has thankfully introduced the term IPFS Mainnet, as a way to avoid this ambiguity:

"Mainnet is a term used to describe the default or "main" network that default settings connect to. This has mostly been "assumed" for the IPFS network, and not really highlighted for anyone other than protocol experts. The technical name for Mainnet is Amino, and there's a blog post from September 2023 that goes into some more details and changes happening." Source: Fission Blog

IPFS Mainnet makes a set of trade-offs that aren't not necessrily fit for every use case. The Shipyard team I am part of is going to great lengths to bring IPFS Mainnet to the Web platform, using Service Workers as a means to both verify hashes on the client side which in turn enables multi-source retrieval. The Service Worker gateway can make any static web app offline ready. You still have to publish it to IPFS Mainnet, which you can do with the help of pinning services or running a node. For example, vitalik's blog which is on IPFS Mainnet can be loaded –albeit still slowly as its still beta software– and cached via the Service Worker Gateway. Once the blocks are cached, subsequent reloads work without internet. The magic is that this is pretty easy to do for any static website.

Having said all of that, I think it's best to think of IPFS as an umbrella for content addressed technologies and libraries rather than one particular combination, like IPFS Mainnet. It's the difference between Lego broadly and specific Lego sets.

I say this knowing that IPFS Mainnet has most of the mindshare and adoption, because I think there's still a lot of unrealised potential for progress in IPFS.

What problems can IPFS solve for local-first today?

It's easy to get lost in theoretical ideas. Therefore, I'll list some of the problems that IPFS solves today that are relevant for local-first software. Not all of these solutions are completely interoperable, and this is not an exhaustive list:

Decentralised web app/site publishing: You take the outputs of a static site build, serialize it with UnixFS resulting in a CID that represents the build, and pin the CID to an IPFS node you run or pinning service. The site can be loaded using a local Kubo node, providing an offline cache for the site. If you don't want to run a node, you can use a public IPFS gateway. A newer approach we're exploring is the Service Worker Gateway, whereby a service worker handles the retrieval and verification.
WebRTC and PubSub with libp2p: WebRTC is complex and leaves signaling to be implemented by the user. js-libp2p has powerful abstractions that simplify signaling in a way that doesn't require trusting the signaling server. Moreover, you can layer GossipSub on top of that as a PubSub messaging layer for ad-hoc mesh networks. I cover this in more detail in the following guide which should be published soon.
Synching data between devices/users: in peer-to-peer fashion using Iroh. Iroh is a new take on IPFS (and therefore not interoperable with other implementations). Iroh provides tools for moving data and synching state directly between devices. It's written in Rust with bindings for Python and Swift (with more coming soon), so it's typically embedded into your application rather than running as a separate binary. Iroh is opinionated, minimalistic, and favours simplicity. The Iroh networking stack is built on QUIC, which means that it isn't supported on the Web (may change?). Iroh has the notion of documents which are mutable key-value stores that can be synched and subscribed to, so it can be thought of as a database. I've heard of rather varied use cases that include cross-device backup, p2p game streaming, and even compute job orchestration.
The Canvas team is taking the WebRTC and GossipSub approach further, as the messaging layer for an authenticated, multi-writer partially-ordered log (based on a Prolly-tree) called GossipLog. One of the many benefits of this approach is that log messages can be synched both directly between peers using a WebRTC connection and also using relays when direct WebRTC connections fail.
Fireproof is an embedded database for collaborative applications. It's built on top of a Prolly Tree implementation and supports multiple connectors for sync. It has some similarities with Canvas.
OrbitDB is a serverless, distributed, peer-to-peer database built on IPFS, Libp2p, and Merkle-CRDTs.
Lucid is an experimental project exploring using CIDs to package Web Tiles — a format for content addressable web sites. Mixes ideas from local-first and verifiability from IPFS.

Identity & keys

Interestingly, almost all of the solutions in the list above use cryptographic key pairs (most commonly ed25519), to sign and identify users/peers/agents. This is no surprise; public-key cryptography enables self-authenticating data structures (aka self-certifying).

Key pairs and identity are intertwined in these technologies, and identity plays an essential role in any user-facing software. The DID spec provides a simple and standardised way for decentralised IDs that can represent these keys.

Fission CTO and co-founder Brooklyn presented a decentralised authorization system called UCAN based on DIDs and JWTs. UCANs are neat because they provide a "trustless" (yet another term for self-certifying) authorization mechanism where capabilities can be delegated without relying on a central server.

Where I think identity gets challenging is UX and standards. Platforms have different key management and recovery mechanisms. Passwordless solutions like Passkeys and WebAuthn are supposed to help with this, but as far as I understand also come with platform lock-in. Meanwhile, most of the crypto space is still reliant on mnemonic seed phrases and hierarchical derivation paths (BIP39, BIP44), though it's slowly changing with account abstraction in Ethereum.

All this goes to say that a seemingly simple building block gets messy and complex as soon as users have to rely on it. Moreover, identity and authentication are hard to change once implemented.

This is an area where I hope to see more progress and convergence around how to best manage user identities and permissions in apps.

Last words

There are so many exciting things happening in software. New tools and paradigms are making new things possible and they don't all have to involve a blockchain.

If you've read some of my previous blog posts, you may have noticed my excitement about the new Web (Web3?) and public blockchains. I must admit that while I still think blockchains are great for some use cases (stateless censorship-resistant money with predictable monetary policy and decentralised name registries like ENS), I don't think they're the be-all and end-all of the new Web. The reality is that crypto and Web3 have made our already financialised life even more financialised and I have misgivings about it.

Local-first is a breath of fresh air and an exciting community of practice to be a part of. But it's just a set of principles that are useful for some problems. I don't need my ride sharing app to be local-first.

At the end of the day, I care about user agency and real-world utility. Elevating user capabilities without gatekeepers.