Going against that grain, this article will show the walls that I’ve run into with CRDTs as I work on BrainFrame. The prior one had a happy ending, while this one does not. Still, it’s worth the exploration to understand why they aren’t the ultimate answer to life, the universe, and everything.
I want you to come away from this article with enough knowledge to understand if you should use CRDTs for your own project. And for those who are just curious, I’d like to welcome you as well. While this post is unavoidably technical, I’ve worked to keep it legible to those who do not live and breathe this work.
Before we dive in: We cover a lot of ground in this article.
The next section, “Some Old Friends Go Shopping”, sets up the example that I’m using through much of this post. It shows a simple grocery list, and gives us the mental model to explore some concepts that will be used by both you and me while we evaluate how best we can use CRDTs.
The following section (“Structured Data and CRDTs”) is all about CRDT data types, and how they work with our example (and, with the library I’m using, how they don’t work). If you’re already familiar with the common CRDT data types, you can skip most of this section. I would, however, recommend picking back up at “Bob never bought dog food - When the tool broke down”. That’s the description of the first failure I had that made me stop and look to see if I was doing the right thing.
The section after that (“I’m not making an overly complicated grocery list. Why do I care?”) is a deeper dive that came out of that insight, and digs into such things as schema vs schemaless YAML frontmatter, and when to choose one or the other. I definitely don’t recommend skipping that one. It’s all about the process I used when trying to make CRDTs and YAML frontmatter work for me in BrainFrame. I wanted you to have this information to help you avoid making the same mistakes I made.
The final section gives my lessons learned, and concrete advice about whether or not you should even use CRDTs with structured data.
Let’s reintroduce Alice and Bob. After they edited the owl out of that document, they discovered they enjoyed each other’s company, have moved in together, and are planning their first ever grocery shopping trip. They need a grocery list. They were even kind enough to give me a copy of it.
store: Trader Mike's
dairy:
- 1 dozen eggs
- 8oz cheese
produce:
- 5 apples
- 5lb potatoes
cereal:
- frosted wheat cereal
- honey nut shaped o's
You might notice that (in the title) I promised some missing dog food, but it’s not on the list. Hold that thought. We’ll get to it as soon as we can.
Attention developers! This subsection is almost entirely to help our non-developer friends understand that example better. If you already know CRDTs and structured data, you should skip it, as it will be redundant for you.
Nobody I know will write out their grocery list looking like that, but I’m doing it this time to provide an example we can work with throughout this article.
Calling back to the previous post, CRDTs are a long-winded way of saying “The devices can always produce the same document, even if two people edit that document at the same time in ways that produce a conflict.” That was all about editing text, just like you might type up in Notepad, or TextEdit, or even vi.
The grocery list above is formatted the way it is to provide an example of something called “structured data”. You’ve already worked with structured data elsewhere: If you’ve ever edited a spreadsheet; if you’ve ever made your own table in your favorite text editor; if you’ve ever made a to-do list; if you’ve ever had your own grocery list. These are all structured data1. Nothing to be afraid of here, you’ve already used it. You just never knew it would be called that.
The list itself is formatted using something called “YAML”2. It’s plain text, just like Markdown is plain text. It has some rules about the formatting, but first and foremost YAML is meant for people, just like Markdown. It’s why YAML and Markdown make for a powerful combination for notes. It might look ugly (okay, no might, it does), but it is very readable by people. You can see which store the list is for, which aisles in the store Alice and Bob are going to visit, and what they’re going to get from each aisle.
What makes this useful for us is that each of these things can be handled differently than normal text. For instance, it’s possible to maintain a collection across devices and have them automatically sync up in ways that just plain text has a hard time with. These special cases allow us to do things that are otherwise very difficult to manage. I’m going to go over a few of the special things we can do next. And when I get to the end of it, I’m going to finally solve the mystery of the missing dog food. Explaining it just requires a bit of understanding first.
CRDTs are well-studied. Because of the properties they have, it’s possible to write code around structured data that does things that are surprisingly easy with them, and difficult without them. We’re going to go over some of the possibilities here, but the Wikipedia article about them has a more comprehensive list.
I’m going to focus on the ones that affect both our grocery list above, and BrainFrame overall. By the time you get through these, you could be forgiven for thinking I’ve forgotten about the overall idea of lists in YAML. I haven’t. We’re going to be treating the aisle listings as sets instead in just a moment. The list idea is what we’d call a “sequence CRDT”, and one example of that is just freeform text. After all, that text is just a list of characters kept in order. To learn more about them, I’m going to refer you to my previous post.
Check out that grocery list again, specifically the very first line:
store: Trader Mike's
The name of the store that Alice and Bob are going to is “Trader Mike’s”. What happens when the edits come in? Alice decides to change the store to “Organics R Us”, while Bob changes it to “Food Tiger”. If we use the same tools from the sequence CRDTs, we’ll wind up with something like “Organics R UsFood Tiger”. Alice and Bob can still have their discussion and figure out the correct store name, but what if they didn’t have to?
We can simply say that the last edit is the one that will always be the one that gets chosen. If Alice made her change at 12:01pm, and Bob made his change at 12:02pm, then we can assume that his change is the one that everything should reflect when everything synchronizes. We can then set the store name to be “Food Tiger” just because his edit was the most recent one.3
In CRDT terms, we call this “Last Writer Wins”, or “LWW”. Generally speaking,
for freeform text, we don’t want LWW. For a single value that should always
match some piece of reality (like the name of the store), using LWW is a valid
choice to make. While this example is a little forced for the store name, other
things matter more, especially for our notes that we want to make. For example,
we might want a status field to always be something the rest of the app can
recognize, like “work-in-progress” or “in-review” or “published”.
Once more back to the grocery list, where we can see things like this:
- 1 dozen eggs
- 8oz cheese
Here, I need to be precise, and that precision can be difficult to see. In YAML terms, this is an ordered list. If code were to redraw this, it should always do so in the same order. For the grocery list, this can genuinely matter. It’s possible (especially depending on how well Alice and Bob know the store) to make the items in the aisle appear in the order that they can be found in the aisle.
Instead, for BrainFrame, we’re going to look at this as a set. This removes the ordered aspect, and adds in another constraint that matters for BrainFrame, but not for the grocery list: A set’s items are always unique. A list, for contrast, can have 12 entries that each say “1 dozen eggs”, but the set can only have one entry that says this. In YAML, they are going to be written down the same way: No difference exists when you look at YAML. It’s all about what the code does with it that makes it different.
Compare and contrast these two: On a grocery list, you will change the beginning of the string to reflect a quantity (“2 dozen eggs” instead of “1 dozen eggs”). For BrainFrame, the most likely use for this sort of thing will be things like tags on the article, which should always be unique. For both of these, a set can actually make sense. So can a list. And that is where the problem truly lies.
This is where intent matters, which is the thing that computers are bad at guessing. I am choosing to treat these ordered lists as sets because I believe that is the intent most of the time. That belief could be wrong, though, and I can’t know until users start working with the program.
The inability to determine the intent from the YAML also means that both of these distinct things are very easily placed under the same conceptual category.
A little more YAML (and its explanation) before we can finally figure out why Bob never bought dog food.
dairy:
produce:
cereal:
These are all the names of the aisles that Alice and Bob are planning to visit when they go to the store. They also happen to represent the most complex concept that we’ve hit so far: Maps (and I don’t mean the thing that has all the roads on it).
Broadly speaking, maps use a string (usually a single word, such as the name of a grocery store aisle) to say what can be found there (such as “8oz cheese” in the “dairy” aisle). We call that string a “key”, and we call what it points to a “value”. You can see it very easily yourself. Here’s the dairy aisle again, just to refresh your memory:
dairy:
- 1 dozen eggs
- 8oz cheese
The dairy aisle contains two items in its set. In this form, it is very easy to see and understand. It becomes harder when you understand that the things pointed to by the key aren’t limited to being sets. The entire grocery list is a map. It has a “store” key, as well as keys for each aisle. Each aisle could be further broken down. For instance, the following would also be valid for the grocery list:
dairy:
eggs:
- 1 dozen large
- 1 dozen brown
cheese:
- 8oz cheddar
- 4oz mozzarella
In this case, Alice and Bob have created a map of maps to make their shopping trip hyper efficient. And they can continue making ever deeper maps. Each value that they point to can be a string (like the store name), a map, or a set of items to buy in that area of the store.
Critical behavior note: In any given map, keys are unique. To use our dairy
example, you can only have one cheese key. When programming, the result of
trying to add a second cheese key will destroy the original value and replace
it with the new value.
Maps are one of the most useful tools that we software developers have. It’s very easy for us to reach for them for many different uses. Sometimes, they feel right at first, and then we find that they’re the wrong tool, and we have to change what we are doing. If we’re very lucky, we find it out early enough that we’ve only wasted an hour. Sometimes, the work can take weeks to unravel.
In my case, the pain wasn’t just the lost time I spent chasing structured data and CRDTs. It was the lost vision. To show why it got lost, it’s time to explain the missing dog food.
Alice and Bob have made it to the grocery store, and they want to make it a quick trip. They split up, and their devices go offline. While they’re each collecting the items from their part of the list, they realize they forgot to add something: pet food! Alice adds the following YAML on her phone:
pet_food:
- canned cat food
Bob adds the following YAML on his phone:
pet_food:
- dry dog food
They’ve each got it in their notes on their devices. Before they actually pick things up from the shelves, their devices manage to synchronize with each other. If everything is behaving correctly, the results should look like this:
pet_food:
- canned cat food
- dry dog food
A new map key was added to both devices. Each device added a set with a single item to that map key. The order of the set on your screen can be reversed from what I show above, but both devices should have them visible in the same order. CRDTs manage this behavior well for structured data.
Plans rarely survive first contact with reality. When I tested this exact scenario, I got a very different result. Here’s what I saw:
pet_food:
- canned cat food
The dog food never made it into the resulting map after the synchronization occurred. I could review the changes that each side was making. I could see the dog food on the one side, and the cat food on the other, and the dog food was simply gone from the result, as if it had never existed.
If Alice and Bob stick to their list, then by the time they get to the pet food aisle, Bob will never buy the dog food. The exact order of operations matters4, so let’s spell it out here:
Remember: Keys must be unique in a given map. So when the code acted the way it did, it overwrote the previous “pet_food” key wholesale. Instead of merging the two sets together (the preferred and arguably correct behavior), Bob’s dog food was relegated to the dust bin.
I lost data. And data loss is the one thing I can’t accept.
And this is the most important question: Can it be fixed? The answer is an unsatisfying “maybe”. This is where the way that the CRDT library is written matters; I’m not sure I could do better than the author on this, which is why the answer is a “maybe”. On the plus side, even if it turns out it can’t be fixed, it can be detected, which is arguably more important for figuring out if your CRDT library is okay.
This is the most technically challenging part of this entire article, but it’s also the most critical for understanding what’s gone wrong.
I hate to tell my non-developer readers to pass it over, but this section is genuinely a lot of geek-speak, and I can’t reduce that. Feel free to move on to the next section. If you do skip, take this one note with you: CRDTs, as a tool, are great at what they do. The specific tool I’m using didn’t live up to those expectations.
When working with CRDTs, everything gets a unique identifier. With the library
I’m using, those identifiers are called “Universally Unique
Identifiers”
(UUIDs). You’ve probably seen them at some point; they look like this:
60c91291-aaad-4f1e-98b0-98f21a603497. Each map key in the grocery list has
one. So store, dairy, and pet_food all have one.
The critical difference is that the others were agreed upon by the devices before everything went offline. This is best described by stating the order of operations explicitly. For this list, it’s entirely arbitrary who did what. What matters is how their devices dealt with the edits.
store key.pet_food key. This generates a new UUID for her pet_food
key.pet_food key. This generates a new UUID for his pet_food
key.pet_food key exist.The poor behavior here is subtle, but important: The library treated the “pet_food” keys as different when deciding to merge (because of the different UUIDs), but then treated them as the same when enforcing uniqueness. That contradiction is what dropped the dog food on the floor.
I am speculating here, but I would be surprised if this isn’t a common bug with implementations doing what I’m doing. CRDTs absolutely require the identity5, and this is an easy bug to miss if you’re not looking for it.
Before moving on, an obvious question you’re asking me right now: How do I check my library for this? I’m going to answer with a bit of pseudocode. You need to translate this for your preferred programming language and the library you are using. Here’s how you can do it in about 40 lines of code:
Once done, you’ll know if you are subject to the same limitations I’m describing. Two notes about this:
In my case, I am looking for it because of BrainFrame itself. I need the things I’ve been describing, but the ways in which I need them can be surprising.
Here’s the surprising thing: Even if you’re not making a grocery list, you are using all those same concepts when you make your notes. I’ll prove it with this sample note:
---
title: Review of The Tell-Tale Heart
author: Edgar Allan Poe
first_published: January, 1843
migrated_to_notion: false
status: work-in-progress
tags:
- horror
- fiction
---
This was a lighthearted romp through
the madness that descended on the
protagonist after killing someone.
I really enjoyed it.
Instead of calling it a grocery list, we call it “YAML frontmatter”. It’s structured text that allows you to make tools that work on your notes. For example, you may want to key off of the status. When that gets moved to “published”, you actually post the review somewhere.
You can see all the elements we discussed earlier: the whole map, the set of tags, the status being a “last writer wins” value.
This subsection is a little more dry than most, and can be summed up very simply: Every field that appears in the frontmatter needs a type, and I’m now explaining how I chose the types. If you’re comfortable already with choosing types, or simply not interested in how to do so, feel free to skip this section.
Thanks to everything you’ve learned throughout this article, you can see how that YAML frontmatter is structured data, and BrainFrame should absolutely honor that structure. I agree with you, and here’s what I would do for that document:
Looking at this, YAML frontmatter practically begs to be treated as structured data. But looking back to the grocery store reveals an issue that makes it problematic.
It’s always easy, right up until it isn’t.
Remember: YAML is for people, first and foremost. Us people are messy, and we come up with new ideas all the time. It is trivial for us to add something to that YAML frontmatter that we had never considered. For instance, maybe we choose to add a link to the Project Gutenberg page for The Tell-Tale Heart so we can download it. That could look like this:
url: https://www.gutenberg.org/files/2148/2148-h/2148-h.htm#chap2.20
While I’m writing the code, I have no idea what keys might be placed in the YAML
frontmatter. The URL is just one example. Maybe the user is doing a survey of
the favorite colors of the authors, so decides to add a favorite_color key. In
order for me to set up the code to handle structured data, I have to tell the
code what that structured data actually is. In software development, we call
that a “schema”.
This lets us say things like “author is a string, handle it like text” and “status is a ‘last writer wins’”. The code then honors those choices, and everything works happily for the choices we have defined, as long as the user uses those things in the same way.
What happens when the user decides to add that “favorite_color” key? Even worse, what happens when they do it on their phone, and then switch to their currently offline e-ink reader and add it there as well (since they didn’t see it yet on their e-ink reader)? We now have two keys, with two different identities, on two different devices. Something is going to get lost.
What about that “author” key? What if we have two documents that try to do different things here: one has only one author, while another has multiple authors? Because we’re dealing with code, we have to make a choice, and a bad choice here mangles what the user actually wants. The author key can’t have two shapes in the code: It has to be either a single string, or a set. And if the user tries to change the shape without a developer updating the code, their data is going to get (at best) garbled. At worst, it could be completely lost, and without any explanation as to why the data got lost.
That’s the real danger: In order to have everything be structured data, we have to commit to a schema that can’t ever cover everything that the user needs.
This subsection is almost an engineering journal discussing the options that got explored and why they didn’t work. I don’t want to cheat my readers out of this information, but I also acknowledge that it might be less interesting to some of you. Feel free to skip forward to the next header if this sort of discussion doesn’t give you any value.
YAML gives us the ability to put structure on the text that goes into the frontmatter. Using that knowledge, we could go for a hybrid approach: Define a list of keys and how we will handle them, and leave the rest alone as user input. If we do that, we can add in keys like title, aliases, url, author, tags, status, and a few others.
Unfortunately, this still is broken from the user’s perspective. As the developer, I will miss more keys than I ever capture, and the user will still be able to create more. What’s more, that same data loss scenario comes back when somebody adds the same key on two separate devices and then synchronizes them. Telling the user “You used the tool wrong, so you lost data, sorry” is never a sentence I want to say.
And with that, the complexity is just beginning. I can say “These are all the keys that are supported by the YAML frontmatter, and no other key will ever be allowed unless/until the code is updated to support it.” By doing that, I’ve just limited an incredibly useful tool into being something that I control and decide, not the user.
I could make two separate front matter sections. The first one will be the one that is all of the structured data that BrainFrame supports, and the second will be the one that the user makes choices about. The problem then becomes “Which frontmatter section is being looked at?” What if the user doesn’t add the BrainFrame supported section, but just adds their own section? They don’t get the benefit of structured data from BrainFrame at all then. And BrainFrame, seeing only one frontmatter section, will have every reason to act as if that frontmatter belongs to the BrainFrame-supported structured data, not the user’s frontmatter.
As a final option, I could make the schema into something that is stored in a note. I could ask the user to take the time to update their schema every time they want to add a new frontmatter field. It’s not impossible, but it’s pretty horrible to tell a non-developer that they have to carefully update a file just to protect their data from being destroyed.
In the end, one surprising revelation is what finally broke me: Comments. Let’s take that sample from above, and add something for us to do later:
---
title: Review of The Tell-Tale Heart
author: Edgar Allan Poe
# need to double-check the first_published data
# and make sure it's accurate
first_published: January, 1843
migrated_to_notion: false
status: work-in-progress
tags:
- horror
- fiction
---
This was a lighthearted romp through
the madness that descended on the
protagonist after killing someone.
I really enjoyed it.
Comments have no structure. They attach to nothing. The comment I added above could be anywhere in the frontmatter. Anything that changes that frontmatter could cause BrainFrame to need to regenerate the entire frontmatter section. The comment could be moved around or even destroyed.
My promise to the user is simple: Your data is protected. I will not destroy it. I will not lose it.
Between the schema issues (especially coordinating that schema across devices which may or may not be accessible when the updates need to happen) and the ability to accidentally destroy the data, I hit the wall I could not overcome.
My vision did not survive the constraints I’d placed on myself. I want to have everything be perfect. I want the user to have the ability to make a schema that suits them. I want to support that schema by having everything be treated properly: Sets, Maps, Last Writer Wins, all of it. I want the user to get all of the possible benefits out of YAML frontmatter they could, fully supported by BrainFrame and CRDTs doing things that other solutions just don’t handle as well. In other words, I want to have my cake and eat it too.
I thought I went into this with my eyes wide open. As it turned out, I was committing one of the cardinal sins of software development. I had my vision. I had a tool. And I got attached to using that tool to make my vision. And when the tool couldn’t support my vision, it hurt in ways that surprised even me. I’m hoping that by sharing these lessons with you, I’ll exorcise some of my own demons and let myself move forward again.
Allow me to shorten these lessons for you now.
In the end, I had to make a choice that I hate, but it’s my way out of the weeds: YAML frontmatter, while it is structured data, is also just text. I’m going to treat it as such, and provide tools to help my users make the best sense out of it that they can should things ever get into a weird state. But I’m not going to drop their data, and that’s the important part.
If you have solutions to what I’ve described, or just want to commiserate about the difficulties, I’m on Mastodon as @pedersen@c.im.
I’ll talk with you soon.
This is not an exhaustive list, don’t treat it as such. These are just some that might be useful to you:
I’m being deliberately a bit handwavy here. Some of these more correctly fall into a category known as “strongly structured data”, while others fall into the category known as “semi-structured data” or “loosely structured data”. Arguably, something like a todo list could even be labeled as “unstructured data”, and would be treated that way in rigorous practice. However, with the way I’m using it here, it fits into the “semi-structured” bucket. There are important technical differences between the three, but this post isn’t about those differences. We’re going to explore why any kind of structured data coupled with freeform text is a problem for CRDTs, so I’m going to continue to ignore those differences for this article. ↩
I hate leaving acronyms unexplained, but this one is basically a joke for computer programmers. YAML stands for “YAML Ain’t Markup Language”. It can do a lot, and the Wikipedia article goes into it in depth. For now, just know that it’s something simple for us humans to read. ↩
When you’re familiar with CRDTs, you’ll catch a simplification here that could throw you off. I’m using the wall clock as a stand-in for the Hybrid Logical Clock, or HLC. An HLC (which is what BrainFrame is using behind the scenes) is how the ordering actually gets determined, not the clock on the wall. I’m using the clock on the wall as an easier way to understand the ordering. HLC is the decider, but that comparison is harder to make in a post like this. If you’re not already familiar with HLCs, it’s worth noting that they do not guarantee a match to the actual time. Frequently, they will line up. However, computerized time-keeping is imperfect, and drifts (or is manually configured to be wrong). Even though HLCs record that time, it’s important to note that they work strictly for ordering, and do not (ever) guarantee that they are actual timestamps. Treating them that way will only make your life more difficult. For still more information, see the article about them. ↩
This order of operations hides something subtle that you might have carried forward from the “Last Writer Wins” topic: The way I’ve phrased things suggests an order based on time. Alice edited, Bob edited, and I’ve kept that ordering because it’s alphabetical, not because of any perceived order. What matters in this case is that the edits were made while both devices couldn’t talk with each other, and then the code had to decide what to do with those edits when synchronization occurred. The code in question did its own internal sorting to determine which operations happened in which order. It could have just as easily gone the other way, where Alice might have not picked up the cat food. Don’t take my arbitrary alphabetical ordering as the canonical order that CRDTs in general would apply things. Your individual CRDT library might use a different clock or a different mechanism for sorting. ↩
Don’t assume that “identity” must mean UUID. It could be a counter, with each character or key getting a number. It could be a UUID. It could be a different type of identifier entirely. This is an implementation detail of the library. It would be good for you to know what your CRDT library is doing under the hood, but CRDTs don’t mandate the use of UUIDs. They’re just convenient for us. ↩
I didn’t cover Observed-Remove in the basic data types because they added length to this post without enhancing anything from the basics section. The idea behind them is that no peer should ever remove data unless it saw that data before trying to remove it. In the bit of pseudocode that brought you to this footnote, if you were to tell peerB to remove “tag1” from the set before the final exchange, the tag should survive. If you were to tell peerB to remove “tag1” from the set after the final exchange, the tag should be deleted. For more detailed information, see Wikipedia and its references. ↩
One single, simple sentence. Perfectly innocuous. Unremarkable even. And yet, somehow, this one little sentence is the introduction to an entire class of problem that has a major impact on BrainFrame (the local-first, own-your-notes app that I’m building). That impact is large enough that I’ve spent literal weeks wrapping my brain around it, and around the solutions for it.
Here’s the interesting part: If you’ve ever edited a shared document, that odd little sentence up there has affected you, too. What should the program do when you and a collaborator both edit the same thing in different ways? Or if you, working alone, edit the same document in different ways on different devices, even offline?
Come with me on this journey into the interior of offline editing. I’ll keep this lightweight. There’s no code discussed in here. This is all about exploring possible solutions and showing why things are the way they are.
Our odd little sentence needs some collaborators. Say “hi” to Alice and Bob. They’re both working on the same document that contains our sentence. Both Alice and Bob think the word “owl” doesn’t really fit their overall theme, so they choose a different animal.
Alice is a dog person, so she changes the word “owl” to “dog”. Bob is a cat person, so he changes the word “owl” to “cat”. As it just so happens, they do their changes at the same time, right down to the millisecond, one letter at a time. And now the program has to make sense of it all.
What should the program do?
Before we dive into anything else, let’s check out how the online world works. You’ve already seen this sort of work happen, and it’s virtually guaranteed that you’ve used it. Google Docs with a collaborator is a prime example of exactly this sort of work. Both people online, editing the same document. Frequently, there is also a voice call where the collaborators are talking through what needs to be done.
The magic in this solution is multi-fold:
You’ve already dealt with these exact sorts of conflicts before. Having the central server referee the results is what happens, and humans are always kept fully in the loop about what is happening in real time. That makes the entire process easy to handle.
One document. One source of truth. Humans controlling every step of the way in lockstep.
Unfortunately, this solution goes against the very nature of BrainFrame. Offline first, local editing, and the lack of a server means that something else has to happen. Fortunately, software developers have been dealing with variations of this problem for decades, and have come up with many ways to work with it and get something resolved. Going over all of those options would be worthy of a book1, so I won’t cover them in this post. I can promise that we have been dealing with it since at least the 1960s2. In one example that comes to mind, a professor in college told the class I was in that one software development group used paint sticks with file names on them. When a developer wanted to update a file, they had to claim the paint stick. And then return the paint stick when they were done. We have better solutions today 😀.
That paint stick anecdote is an example of something called “pessimistic locking”. On the surface it seems like a viable mechanism. Why not just prevent somebody from editing the same file at the same time?
What happens when the edits don’t hit the same area? Alice is working on a section talking about the colorful owl. Bob wants to work on a very different section, and that section is just a table of contents for the whole document. Alice has the stick, so Bob can’t make his edits even though there’s no reason to prevent the edit. And what happens if Alice goes on vacation, or loses the stick, or it gets broken somehow? Unless that stick gets replaced correctly, nobody else can ever edit that file.
That sounds like it wouldn’t be a problem in today’s world, but what happens if the paint stick is being held by a phone, and that phone gets dropped in a lake? Without some work by somebody with technical skills, that file is locked forever. Bob can never update the table of contents.
There is a variation called “optimistic locking” which addresses some of these problems, but even it has limitations. For optimistic locking, the file would get checked on save. The program basically asks “I’m about to overwrite this file with new contents. Did somebody else update the same file while I was editing? Or is everything good to go still?” The biggest problem with optimistic locking is the need to have a second view of the new file. And somehow, the user has to be able to tell the differences between the original file, the version somebody else saved, and the new version of the file they are trying to save.
It’s a lot to ask of a human: Compare three things and figure out what changes need to make it into the final results. Software development has a huge toolkit built around this exact problem, and it’s still messy. For something like “The colorful owl sat” it’s easy. But what happens when the changes are spread out over a thousand lines? How can the human in the loop resolve this in a meaningful way and not lose entire days just handling a simple line-by-line comparison?
This is where a pattern is beginning to form: All the solutions sound easy, right until they aren’t.
Software developers reading along have already picked up on this next section because they live it every day. We call these patterns “diff” and “merge”.
Broadly speaking, there are two kinds of changes: conflict-free and conflicted. Using our example above, what happens when it’s only Alice editing the file? She changes “owl” to “dog”, hits save, and that’s the end of the story. This is a conflict-free change, and it’s extremely easy to manage. If Bob makes a change to the table of contents, he’s not even editing the same section of the file. We now have tools that make both of these changes conflict-free (for instance, git), so we software developers have, by and large, abandoned using locks for writing our code. It’s safe to say that conflict-free changes are almost boring at this point.
Conflicted changes are where it gets more interesting again. Go back to our original example: Alice changes “owl” to “dog”. Bob changes “owl” to “cat”. Unlike the real-time world of always-online, there’s no one to talk to offline. In the world that BrainFrame lives in, that’s not viable. A human needs to sit down, review the changes, and make the choice to determine what edit wins. Entire tools and websites (like GitHub) exist to make this process more manageable. As a developer, we go to the central source of truth and make a choice. Once we have made that choice, all the other developers have to grab that choice into their code.
The tools we have make it manageable, but they are decidedly not friendly tools. Asking non-developers to master these tools is being unkind to them.
What happens, though, when there is no central server? What happens when Alice and Bob make their edits at the same time, right down to the millisecond? It’s not a good idea to ask the user to use tools like GitHub for what is free-form prose (the heart of every note every user makes). It’s not a good idea to present a separate note into their notes collection calling out the conflict. The user has to spot it, and that is something very easy to miss.
The common thread for all of this is the decision: The user is always involved. The computer can’t make the decision, because it doesn’t understand the intent of the user. Without that understanding, it can’t do any resolution. And without that resolution, the user has to step in and assert control over the process. Sometimes, this is required. But most times, and especially for free-form text, it really isn’t.
How can we let the computer do the work for us, and just tell us if/when we need to be involved afterwards?
After all of the above, you might be feeling hopeless. I know I did. I know what I want, but I didn’t have anything to anchor it off of to actually get what I want. Not until I found out about Conflict-free Replicated Data Types, or CRDTs. These were first defined in 2011, and have been refined greatly in the many years since then.3 The technology itself is old, but for me, it is very new. I’ve only been learning it over the past couple of weeks.
That mouthful of a name is a long-winded way of saying “The computer will always figure out a merge path, and will not trouble the user with the need to deal with merges.” In the next section, I’ll go over why this isn’t perfect, but for now, for BrainFrame? CRDTs cover over 90% of all the cases for me very well.
Let’s build up what we’re doing, and how CRDTs handle them, one step at a time. As I promised in the beginning, I’m not going to post code here. This is all about what the results of using CRDTs are, and why they help.
Alice types up her note: “The colorful owl sat.” She saves it. Comes back a day later, and changes “owl” to “dog”. For this case, we don’t need any special mechanism. Your favorite text editor handles it without any stress to it or to you.
Alice and Bob share the laptop. Alice logs in, makes a change to the file. Bob logs in, makes a different change to the same file. Again, nothing special here. All of the cases so far are manageable with your favorite text editor.
A naive approach to this would say that there’s really no difference between this and the first case. Alice can only edit in one location at a time, so the changes are always going to be visible. That approach assumes an always-online world, even without a centralized server. What happens if Alice’s phone runs out of battery? If she had changed “owl” to “dog” on her phone beforehand, we have an edit that her laptop doesn’t know about. She continues editing the original document on her laptop, waiting for her phone to charge up. Once it does, she turns it on, and the synchronization process begins. The “owl” becomes a “dog” finally. But what if she changed her mind, and decided to turn the “owl” into a “cat” while waiting for her phone to charge up? What happens then?
Alice has a laptop, Bob has a phone. They are editing the same note, at the same time, and staying in sync. There is no central server, the two devices have to figure everything out and present the results to the user in a way that makes sense. Possibly even more importantly, the two devices have to present the same results to Alice and Bob without the devices “arguing” about which one of them is right.
Remember: Alice changed “owl” to “dog”, and Bob changed “owl” to “cat”. How do we decide on the result in a way that allows Alice and Bob to find and fix the conflict?
This is the heart of the problem. Everything else is a special condition on top. How do two devices receiving concurrent updates from each other make sure they’re giving the users the same result at the same time without some sort of arbitration mechanism? How do they converge on a result they give to Alice and Bob?
CRDTs do exactly this. I’m going to be a bit handwavy here, since I promised no code. What CRDTs do, though, winds up feeling like magic, especially when there’s no conflict. Alice changes “owl” to “dog”, Bob edits the table of contents. Both devices synchronize. This is the most boring possible case for all of it: The changes simply land where they are supposed to.
The whole process is transparent to Alice and Bob. No having to go to a central source of truth and figure out the conflict and then make sure everybody gets it. For Alice and Bob, the work is handled automatically by the computers, the updates land, and everybody is happy.
The changes are anchored in the text in such a way that even if one of them deletes a major section, as long as they don’t conflict with each other, all changes will come out matching Alice and Bob’s intent. The document will remain coherent and complete.
It might not be magic, but it certainly will feel like it. Documents get synchronized across multiple devices with multiple users and everything continues to make sense.
Or does it?
CRDTs provide a way for the computer to determine what should land, in what order, when document edits collide. I cannot overstress the importance of this enough: The mechanism only guarantees you a result; it does not guarantee a meaningful result. For that, we still have to rely on the human in the loop. What we can do, though, is make it easier than the tools that programmers have to use. Prose doesn’t have to go through the same process that code does, so if a mistake lands somewhere, the fix is easier to manage.
Make no mistake, though: Conflicts are a real problem, and not even CRDTs solve them entirely. For that, let’s lean into our example from the beginning: “The colorful owl sat.” Alice changed the word “owl” to “dog”, and Bob changed the same word to “cat” at the exact same millisecond that Alice did. Above, I’ve asked you to consider what happens, and how other systems try to solve it.
CRDTs solve it differently. Instead of picking a winner, CRDTs let both edits win, simultaneously. All by themselves, CRDTs will produce a result that is undeniably gibberish. “owl” will become something like “cdaotg”. Look through that mess, and you’ll see the words “cat” and “dog” mixed together. The reason for this is that additions to the note are prized above all else. If somebody added something, then the result should have all of what they added. Since Alice and Bob were typing at the same speed at the same time, their results got mixed together at the same spot in the document. While the gibberish is, of course, bad, it has one property that is good: It is consistent. Each device that tries to view the note will get the same gibberish.
Have hope: All is not lost. This is a well-known problem, and it has a solution that makes the gibberish more legible to humans. That solution is called Fugue. When we apply that solution, the letters of the word stay together. The result of using Fugue on these same edits will be either “catdog” or “dogcat” (there are some sorting rules that come into play that I won’t go into for this post). While it’s obviously wrong, it’s also something that us humans can reason through and fix. Either Alice or Bob will make updates to the note, removing what they collaboratively decide is the wrong word, and the note will make sense.
Where the hard part comes in is making sure that the conflicts are brought to the attention of Alice and Bob. If they’re working with an entire collection of notes, and these edits conflict, the application needs to notify them about the conflict and direct them to its location. Then the human in the loop can fix the problem.
There’s something even stranger hiding in our example: “The colorful owl sat.” Alice decides to change the word colorful to the British spelling: “colourful”. Bob decides to delete the line entirely. This is the most counterintuitive part of CRDTs. Before I tell you the answer, can you guess it? Take a moment and think about it. What will happen?
The answer will surprise you unless you’ve already spent time studying CRDTs. The algorithms in this case are heavily focused on preserving every single addition after all the deletions happen. If somebody took the time to add something, we want to make sure that addition gets preserved. That’s why the only thing left in that example is the single letter “u”. No, that’s not a mistake. Alice added a letter. Bob deleted every letter in that line except her addition. Alice’s newly added letter survives.
It might seem like this is a problem that the computer should be able to solve, but it is actually impossible. The computer doesn’t know what Alice and Bob want. It can’t discern intent from the text of the note. It can only direct their attention to the conflict and let them decide what the correction should be. Someday, we may be able to hook our brains up directly to the computers and express our intent. If that day comes to pass, then the computer can solve it for us.
That’s not the only cost of CRDTs. CRDTs have hidden bookkeeping costs. They have to track every single change to every single character across every single device. Those changes can’t be discarded until every device has checked in and confirmed it has the updates for that note. If a device suddenly winds up at the bottom of a lake, we have to provide additional housekeeping tasks that allow Alice to tell the computer that a device is gone and won’t come back. Otherwise, the collective changes for the note will grow forever. A single note that is only a few kilobytes in length could have its change history occupying hundreds of kilobytes (even megabytes) otherwise.
CRDTs make compromises, the same as every other solution we have. Those compromises keep most of the cases running perfectly most of the time. As long as we make sure that Alice and Bob can keep their eyes on enough information to make those compromises acceptable, they work well.
Of course, this is all about text. Binary files (like graphics, PDFs, and ePubs) are an entirely different story. And that story is part of BrainFrame’s overall story, too.
We’ve covered a lot of ground here. We’ve got history, we’ve got software development, we’ve even covered a few topics that most people outside of software development have never even heard of (“pessimistic locking” anyone?). All of this material is a way for me to communicate why I’m making the choices I am for BrainFrame.
CRDTs are a tool that takes away most of the frustration of shared offline editing with no central server. Unless you’re a developer, you’re not likely to want to deal with locks and merges and conflict resolutions. Scratch that last one: Not even developers want to deal with conflict resolution; it’s pretty horrible. That doesn’t change the value of CRDTs.
For you, the goal is to provide a set of markdown files that you own, you edit, you explore in your own way. To do that, and to provide the multi-device experience that has come to be expected in the current age of computing, BrainFrame has to do some background work. I’m already working on that, and I’m making sure that the work is isolated from your notes. You will be able to use any editor to make the files (yes, even Notepad/TextEdit/vi if you so choose), and BrainFrame will do the work to make sure all your devices get it.
As was shown above, CRDTs aren’t magical, no matter how much we all wish they were. Despite the ways in which they can miss, they provide escape hatches for us. We can detect when things look like they weren’t handled smoothly. We can make sure to bring those to the attention of the user of BrainFrame. We can provide the right assistance, at the right time, to make sure that notes stay coherent. It’s a big bet, but it’s one that I can (and will) deliver.
Another question some of you might be picking at: What about binary files? Their answer is more mundane: The entire file has to be edited as a single unit. The actual send between devices might be done in individual blocks and reassembled, but the edit can never be at the individual character level. PDFs, ePubs, and graphic files do not survive having their internals edited, and quite likely will have their native editors and viewers simply put up an error message saying that the file is corrupted. Sadly, we don’t have the magic for handling them properly yet. That’s okay for us, though, since BrainFrame is about providing a means to annotate and link to those files, not to make edits directly to them.
Other considerations do exist, and I’m not going over them in this article at all. Things like “How do other devices actually get those changes?” are very different questions, and are likely worthy of multiple articles.
“The colorful owl sat.” Four innocuous words. Three changes, “dog”, “cat”, and British spelling. Six words in total. And all of it turned my world upside down.
You don’t have to be a software developer for this to have turned your world upside down either. You can see the problems. After this article, you can even talk about them, maybe for the first time. And that is my true hope: Not that I’ve turned your world upside down, but rather that I’ve given you something new to think about. And if you are a software developer, perhaps even a new problem to help solve for the world.
Go forth, and edit your notes. My job is to make sure those edits work. I’m on it.
If you’d like to have a conversation with me about this article, you can find me on Mastodon as @pedersen@c.im. I look forward to it!
Amusingly, this doesn’t seem to exist at this time. The history is scattered across chapters, papers, and articles. The topic is still book-worthy, and will have references to tools that are considered laughably primitive today. I’m not the person to write it, though. BrainFrame demands too much of my time for me to make an effective author for the topic. ↩
The first program to formalize this whole process was called SCCS, and was released in 1973. The problem itself is, of course, older. The code wouldn’t have been written if the problem didn’t exist. Considering the first multi-user operating system looks to be CTSS (released in 1961), it’s safe to assume the first conflicts were being felt by users around the same time. ↩
A similar technology known as “Operational Transformation” takes a different approach. First defined in 1989, it was later adopted for use in Google Docs. While there is no actual requirement for using a central server to be successful with Operational Transformation, the complexity involved in getting it right (especially when CRDTs are much easier) makes this option a non-starter for BrainFrame. ↩
Imagine being able to view your notes as standalone items as well as in the context of the original source. You could review them and see where your thoughts about a topic actually came from, and understand them better.
Imagine that application working across all three major desktop platforms (Windows, Linux, and Mac), as well as the two major cell phone platforms (Android and iOS). Imagine all your data being synchronized across them as easily as you starting the application on a device.
Imagine all of this data being stored in as neutral a format as possible (your notes in plain text, with the links to the original source being stored in a way that would eanble you to find your way to the ideas without using this application). Imagine your data beings yours, first and foremost, stored on your devices rather than in the cloud.
Imagine the cloud being an option for storing your data.
Imagine this being available on a standalone device as well as all of the above.
All of these imaginings are what brainframe is meant to become. A reader for these formats. A way to take the information on the go. A way for you to read, to explore, to take notes, to connect ideas that were previously difficult to connect.
Over the next few weeks, I’ll go over why the current applications people use for this sort of work don’t cover everything. The applications are amazing, make no mistake, but they are incomplete implementations of this idea. Obsidian, Notion, Roam, OrgMode, and others all fill this idea to various degrees, but none of them implement it fully.
I am still teaching myself new technologies to help me make this. As a result, the weekly posts I will make here will not be about the creation of Brainframe directly, but rather about my thinking as I move forward with it. I hope you’ll join me on this journey. It looks to be an interesting trip.
]]>