Read path architecture#
This page answers: “Which module owns each step between a dataset’s bytes
and its pixels, and what must stay true there?” It is a map, short on
purpose: each step names its module and entry points (all under
cellucid/assets/js/) and the invariants that module’s header and tests
hold. The long-form description of sources, caches and failures is
Data loading pipeline and caching; which cells a subsample keeps is
Subsampling specification; drawing is
Rendering pipeline (WebGL) and performance notes. In the code tree,
assets/js/ARCHITECTURE.md has the layers, the glossary and a “where to
change X” index, and each folder has a short README.md.
At a glance#
Audience
Developers and coding agents changing how data is read, priced, kept or drawn.
Time
15 minutes.
Prerequisites
The path#
page thread reader worker (one per open dataset)
─────────── ─────────────────────────────────────
1 source data/*-source.js,
data/sources/ (provider,
identity, remote-file)
│ open(source, {cellSubset})
▼
6 client data/anndata-reader-client.js ──postMessage──▶ data/anndata-reader-worker.js
│
2 bytes formats/io/: byte-source → byte-reader
│ (+ kept-parts, range-cache, link, hosts)
3 containers formats/{hdf5,zarr,zip,rds,text}
│ → store model (formats/store.js)
4 AnnData data/anndata/open.js → views/ | transcode/
│ → AnnDataReader (reader.js)
5 share planner subsample-plan.js → views/cell-subset.js
│ gene sample, column indexes, graphs
◀──── payloads (typed arrays, transferred) ──────────────┘
7 page state app/state/ (DataState, field managers, gene-sample.js),
app/open-cells-resize.js
8 renderer rendering/ (alpha layers, read-cell layer, HighPerfRenderer, LOD)
Cross-cutting, on both threads: memory (data/memory-budget.js),
cancellation (utils/abort.js), progress and logs (formats/io/activity.js,
data/read-activity-log.js, data/transfer-log.js,
data/opening-progress.js).
1. Sources: what is opened, and which version#
Owns |
Module |
Entry points |
|---|---|---|
What a file or folder is |
|
the format registry (mirrored byte for byte as |
One class per way a dataset arrives |
|
|
Link providers |
|
|
A remote file described without downloading it |
|
|
The identity and version of a file |
|
|
A link’s public form |
|
|
Invariants: only a file whose identity names a version is kept across visits; credentials never enter an identity, a log, a session or a URL the page shows; feature state never goes into URL parameters.
2. Bytes: requests, the link, kept bytes#
Owns |
Module |
Entry points |
|---|---|---|
Where bytes come from |
|
|
Turning reads into requests |
|
|
The measured link |
|
|
Each web host |
|
|
What a host delivered on earlier visits |
|
seeds a link before it is measured |
Bytes kept across visits |
|
|
Bytes other readers of the page fetched |
|
|
Order of a read: the byte reader’s own memory, then kept parts (this machine’s disk), then the page’s range cache, then the host.
Invariants:
A gap between two ranges is fetched only when it saves time on the link as measured now; plans never merge across bytes already kept or held, and a read joins a request in flight that carries its bytes.
Request shaping uses the shortest round trip; prices of reads use the round trip under load (
loadedRoundTripMs) and no less than the host’s measured cost per request.Until the link is measured, the source’s prior stands (a local file is a disk; a web host is read ahead only once measured).
3. Containers: the store model#
Owns |
Module |
Entry points |
|---|---|---|
The model every format is opened as |
|
|
HDF5 (lazy tree, chunk indexes, read plans) |
|
|
Zarr v2/v3 (folders, zip, web hosts) |
|
|
ZIP, R serialization, MatrixMarket |
|
|
Decoding off the reader thread |
|
|
Invariants: a value returned by read() belongs to the caller; an unloaded
group throws StoreNotLoadedError rather than answering empty;
storedBytes(selection, {toFetch: true}) counts only bytes a read would
fetch now, and every time estimate prices that; every read made through an
open Zarr store or HDF5 file runs within its lifetime
(utils/abort.js withinLifetime), so close(reason) stops each one
still under way – its fetch, its wait to retry, its place in the queue –
whoever asked for it, background reads with no signal of their own
included, each rejecting with reason (the reader passes why it moved on,
AnnDataReader._superseded; by default a closed cancellation).
4. AnnData: one reader for every format#
Owns |
Module |
Entry points |
|---|---|---|
From a source to a store root |
|
|
Formats presented as AnnData in place |
|
|
Streams converted once into a kept Zarr store |
|
|
The reader |
|
|
Gene columns and cell rows by layout, and their prices |
|
|
Invariant: a format-specific path is a view or a transcoder, never a branch
in reader.js.
6. Worker and client#
Owns |
Module |
Entry points |
|---|---|---|
The worker’s API over one reader |
|
one handler per client call |
The page’s end |
|
|
What is read first, and never twice at once |
|
|
Genes straight to an analysis worker |
|
|
Payloads made in the worker |
|
coordinates, forward adjacency, codes ready to draw |
The adapter the app talks to |
|
Invariants: the page never blocks on a file; typed arrays cross by transfer;
failures cross in one wire format (utils/error-wire.js); cancellation is
utils/abort.js on both sides; a reader that leaves ({leave: true}, a
source disconnected or cleared) closes its store at once, so no read it
started outlives it, and a link, remote or viewer-host opening that fails
cancels the requests it asked for beside the one that failed.
7. Page state#
Owns |
Module |
Entry points |
|---|---|---|
The state coordinator |
|
|
Field loading and residency |
|
|
The gene sample on the page |
|
|
Which cells are open, across boundaries |
|
|
Changing the open cells in place |
|
Invariants: inside the app per-cell arrays are in open-cell positions; everything that leaves (sessions, exports, viewer protocol, Python and R) names cells by original position, converted only through the cell identity. A cell not read for a gene holds NaN (no value), never a zero.
8. Renderer#
Owns |
Module |
Entry points |
|---|---|---|
What a colour’s alpha byte means |
|
|
How unread, zero and read cells look |
|
|
Points, LOD, culling |
|
|
Invariants: every surface that draws cells (viewer, figure export, smoke) takes its layer colours and alpha meanings from these two modules; unread cells are drawn first, so every read cell shows above them.
Why it is this way: decision records#
Each step’s dated decisions, the measurements per engine that justify them, what was tried and reverted, and the open items: Decision record: decoders and codecs (step 3 decoding), Decision record: link model and request planning (step 2), Decision record: partial reading (step 5), Decision record: graph edges at scale (graphs in steps 5 and 8), Decision record: cell exports, Decision record: served stores and notebook widgets (served sources in steps 1 and 2), Decision record: memory model (counted memory, every step), Decision record: rendering layers, the gene-sample look and levels of detail (step 8).
Where the read path is tested#
Step |
Tests (in |
|---|---|
Bytes |
|
Containers |
|
AnnData and share planner |
|
Page and renderer |
|
cellucid/scripts/efficiency-floor/ measures every source × format × share
against the floor the link allows.
Next: Codebase map and entry points for the rest of the tree.