kappa5 sparse data flow

kappa-core@5 introduces "sources" which are hooked up to views. This decoupling makes sparse patterns possible, but the patterns for how to rig up the data flow for an application are not necessarily obvious.

Further complications arise when you consider how to query for data which may or may not exist locally or with peers on the network. Where should the api endpoints reside for local queries vs requesting information from the network? It's probably insufficient to have a cache fallback layer like with more server-oriented applications because latencies can be much higher and you can never be sure when a request is "done". Updates and data could stream in over a long period of time from devices that may have been offline for a while.

Here I propose a model that borrows from the "unidirectional flow" architecture from web and game development.

    .--------------------------------------------.
    |                                            |
    v                                            |
[source SA] --> [view VA0] -------.              |
    |                             |              |
    |                             v              |
    '---------> [view VA1] --> [ui UA0] --> [query QA0]
                     |            ^              ^
                     |            |              |
                     '------------+--------------'
                                  |
    .-----------------------------+--------------.
    |                .------------'              |
    v                |                           |
[source SB] --> [view VB0] -------.              |
    |                             |              |
    |                             v              |
    |---------> [view VB1] --> [ui UB1] --> [query QB0]
    |                |                           ^
    |                '---------------------------'
    '---------> [view VB2] --> [ui UB2]

    .--------------------------------------------.
    v                                            |
[source SC] --> [view VC1] --> [ui UC0] --> [query QC0]
                     |                           ^
                     '---------------------------'

Sources provide raw sparse feed data to views. The views calculate indexes on the feed data for the user interface to display.

The arrows in this diagram are continuously moving data through the layers live as it's downloaded from the network and as the user interface sends queries to fetch more data.

The query boxes in this diagram are components that ask peers which feeds and sequences are relevant to an application-specific query request. A query request could ask about a key or a range of keys in a key/value store, all the records inside a bounding box for a geospatial database, or all the messages in a channel in a time range for a chat app.

Sometimes the UI will not be hooked up to a query. Other UIs will drive queries that cause the sources to pull more data for the views hooked up to that UI instead. Often a UI will be hooked up to views that pull from different sources.