Progress in Derivation Land

Derivation is a set of libraries I've been working on to make collaborative web applications and things have progressed!

There have been three foundational changes to talk about:

Changes

1. Building the graph as you evaluate it

In earlier versions of derivation. You couldn't create new ReactiveValues while the graph was stepping, but now you can. The constraint is that you can only depend on values that have already been computed in the current step. In practice, this isn't a problem unless you are trying to send values 'back in time' to where you want to use them using mutation.

Earlier versions of derivation acted like a spreadsheet. Now, it's a spreadsheet where you can construct new cells dynamically.

2. Dirty-tracking

Earlier versions also computed everything every step. Now, there's dirty tracking so that we only evaluate nodes when their dependencies have changed.

I'm sure it sounds very wasteful to recompute everything ever step, but if your graph is coarse, it's likely that everything will need to be recomputed anyway and you just wasted time on book keeping to figure that out.

I made this change though, because, increasingly, I want to do fine-grained reactivity, where this pays off. I've also been putting this off, because I wanted to keep things simple while I experimented.

3. @derivation/composable

Earlier versions of derivation also included relational operators (similar to DBSP). I've been hitting the limits of this data model, however, and experimenting with with something more powerful, which I'm now publishing under the @derivation/composable package. The relational operators now exist in @derivation/relational.

With @derivation/composable, you'll be able to write very straight-forward, functional looking code using your favorite immutable data-structures, but with the ability to make small adjustments to the inputs and see the outputs update efficiently.

It's a magic trick which is fantastically difficult to pull off, but which exposes an incredibly expressive programming model.

Miscellany

Aside from these foundational changes. There are also new libraries:

• @derivation/rpc extends your derivation graph over websockets and into browsers,
• @derivation/pg allows you to source your derivation data from postgres,

What this means

Although it's not mature enough yet to use in production, derivation is quickly and quietly turning into a formidable toolkit for writing web applications.