Analysis architecture#

This page documents the Cellucid Analysis subsystem: how analysis UIs are registered, how data is queried, which compute backends exist, and how the module avoids memory/performance degradation during long sessions.

It is written for contributors adding:

a new analysis mode,
a new plot type,
a new compute operation,
or changes that affect analysis ↔ highlights ↔ sessions interactions.

At a glance#

Audience

Computational users: read “What analysis depends on” + “Troubleshooting”.
Developers: read fully and then follow Extension point: add an analysis mode for implementation steps.

Time

30–60 minutes

Prerequisites

State (DataState) and events (highlights/pages and events)
Data loading pipeline and caching (var fields and lazy loading context)

Where analysis is initialized (not a normal UI module)#

Unlike most sidebar modules, Analysis is created directly in:

cellucid/assets/js/app/main.js

It looks for a DOM container:

#page-analysis-section

Then constructs:

createComparisonModule({ state, container })
- implementation: cellucid/assets/js/app/analysis/comparison-module.js

For debugging convenience, the instance is stored as:

window._comparisonModule

Note

Analysis has its own UI architecture under cellucid/assets/js/app/analysis/ui/. It is not orchestrated by cellucid/assets/js/app/ui/core/ui-coordinator.js.

Mental model: “analysis runs on pages + highlights”#

Most Cellucid analyses are framed around:

Highlight pages (independent groups of highlighted cells)
Visibility (filters/outliers)
Variables (categorical obs, continuous obs, gene expression)

The comparison module listens to state events:

page:changed (pages added/removed/renamed/switched)
highlight:changed (membership changes)

Code:

cellucid/assets/js/app/analysis/comparison-module.js

This is why highlight pages are treated as first-class state: they are analysis inputs, not just UI decoration.

Code map (analysis directory)#

Analysis lives under:

cellucid/assets/js/app/analysis/

Key entry point:

analysis/comparison-module.js
- orchestrates the analysis UI and binds it to DataState. Modules import each other directly; there is no barrel index.js for the analysis tree.

Subdirectories (high-level):

analysis/core/: plot plugin contract, validation, the plot registry
analysis/compute/: compute operations and their Worker and CPU backends
analysis/data/: data layer and streaming gene loader
analysis/stats/: statistical tests, corrections, result formatting
analysis/plots/: plot infrastructure and types
analysis/ui/: UI manager, analysis-type UIs, floating windows, shared UI components
analysis/shared/: plot theme, debug utils, cleanup utilities, the run tuner

Core architectural pieces#

1) `ComparisonModule` (orchestrator)#

Responsibilities:

registers analysis “types” (Quick/Detailed/Correlation/DE/Signature/Marker Genes, etc.)
manages shared configuration (currentConfig)
coordinates page selection and mode switching
triggers plot creation and manages plot lifetimes
restores analysis windows from their exact descriptors (the analysis state domain)

Code:

cellucid/assets/js/app/analysis/comparison-module.js

2) `DataLayer` (unified access to state + datasets)#

The analysis module needs a safe, consistent way to:

extract per-cell values for obs/var fields: a continuous page holds the field’s own numbers, a categorical page its cells’ category codes at the field’s width with categories and missingCode (counts keyed by label are published with the codes)
read whole gene columns for computations (readGeneColumn(s)): a gene’s sparse entries from the source, or a loaded field’s values, never installed in field state
handle missing values
respect filtering/visibility/highlights
cache page data by page version

This is handled by:

cellucid/assets/js/app/analysis/data/data-layer.js

3) Compute backends (Worker / CPU)#

Compute is structured around “operations” with capability metadata: which operations can run in a Worker and which can run on the main thread. Every operation runs the same function on either backend, so where it runs never changes a number. The operations are:

correlation (COMPUTE_CORRELATION) and differential expression (COMPUTE_DIFFERENTIAL);
kernel density (COMPUTE_KERNEL_DENSITY): a density-plot curve of more than 3,000,000 kernel terms is split by curve point across the worker pool, and every point runs the same loop as the single-thread sum, so the curve is bit identical; COMPUTE_DENSITY_PLAN filters a page of 200,000 values or more, picks its bandwidth and returns the curve or the plan the sum is split by;
box statistics (COMPUTE_BOX_STATISTICS, pages of 100,000 values or more) and violin density slices (COMPUTE_VIOLIN_DENSITY, pages of 20,000 or more), which mirror Plotly’s own calculation. For a violin the worker also returns the calcdata entry Plotly’s box calc would build, which the vendored Plotly takes instead of building a point record per value (see the header of cellucid/assets/external/plotly.min.js);
histogram bins and samples (COMPUTE_HISTOGRAM_SAMPLES, 200,000 values or more over all pages);
the summary statistics and the statistical tests a Detailed or Gene signature view shows (COMPUTE_SUMMARY_STATISTICS, COMPUTE_STATISTICAL_TESTS, 200,000 values or more over all pages);
order statistics (COMPUTE_ORDER_STATISTICS): the median and quartiles of Quick Insights and the median of signature scores, for 500,000 finite values or more.
hierarchical clustering (COMPUTE_PAIRWISE_DISTANCES in row blocks across the pool, then COMPUTE_AGGLOMERATION in one worker, compute/hierarchical-clustering.js), always in the workers;
per-cell CSV files (COMPUTE_CSV_EXPORT, compute/csv-export.js): each worker writes a range of rows as a Blob, joined after the header.

Below those sizes the main thread computes the result directly, because posting the values to a worker would cost more than the work. A payload posted without transfer is copied on the main thread, so the worker pool copies at most 96 MiB per task and posts the rest in tasks of their own (DISPATCH_CLONE_BYTES_PER_TASK in compute/worker-pool.js): no dispatch holds the interface for more than about 10 ms. A worker that scanned a page also reports its finite values, which the main thread adopts instead of scanning the page again, and categorical counts are taken once, as the data layer builds a page’s values (analysis/shared/category-counts.js). Loops that must stay on the main thread (gathering a page, aligning two variables for a scatter plot, signature scoring from sparse gene columns) yield to the interface between 12 ms slices. Marker-gene discovery dispatches its own MARKERS_* messages to the same worker pool. There is no GPU analysis backend.

Key files:

cellucid/assets/js/app/analysis/compute/operations.js
cellucid/assets/js/app/analysis/compute/operation-handlers.js
cellucid/assets/js/app/analysis/compute/compute-manager.js
cellucid/assets/js/app/analysis/compute/worker-pool.js
cellucid/assets/js/app/analysis/compute/data-worker.js

Design goal:

select one declared, available backend before an operation starts; execution errors are reported directly and never switch to another backend.

3b) Gene downloads#

Every gene and observation-field load goes through one download scheduler (cellucid/assets/js/data/fetch-scheduler.js), entered from app/state/managers/field/loading.js:

Priorities. A gene the user chose or the field a view is coloured by (high), work the user started and is watching, such as a gene signature or a correlation (normal), one of a long run’s genes (bulk) and a prefetch on intent (low). high and normal reads start at once; while any is in flight, no new bulk or low read starts, so a gene chosen during a 5,000-gene run does not share the connection with a dozen of the run’s reads. A read joined by a more urgent caller (a prefetched gene the user then chooses) is raised. Waiting reads start in priority order (bulk before low, each kind first come, first served), no more of them at once than requests the dataset’s source answers at once, as its reader reports it (setSourceRequestConcurrency): six to an HTTP/1.1 host until a reader says otherwise, the host’s streams over HTTP/2 or HTTP/3, and no bound for a source whose reads cost no request (a local file).
Reads ahead, straight to the workers. A run that streams genes (markers, DE, the gene-moments pass, the dot plot, UCell: analysis/shared/gene-stream-runner.js) hands them to the workers in tasks of several genes, and reads the tasks queued after the ones being computed, as their genes’ sparse entries (never installed in field state). The reads go over the run’s gene route (analysis/data/gene-route.js, data/gene-delivery.js): one read serves a task on each worker, the reader posting each task’s genes to the worker it runs on, and the page hears only their bytes, so no gene value crosses the main thread. How many genes a task takes, how far ahead, and how many tasks the workers hold, is decided by the run’s tuner (analysis/shared/run-tuner.js) from the worker count, the run’s genes, the analysis memory budget (the analysis share of data/memory-budget.js less the per-cell state the run keeps in every worker: compute/worker-pool.js analysisRunBudgetBytes), each read’s time and bytes and the link the reader measured: sixteen tasks per worker over the run, held so the tasks in flight fit half the budget: at each gene’s densest form until a read is measured, then at the genes’ bytes as read, sized again at each read and each shrink of the budget (held at the densest form, a 10M-cell differential expression of two groups went one gene a task, 30,000 tasks and page messages); the read-ahead covers the tasks consumed while one task’s genes are fetched, doubles while workers sit idle waiting for reads, halves when throughput falls or the budget shrinks (a refused allocation, a larger working set, another page of the origin), and buffered genes hold at most 30% of the budget. The tuner never decides network parallelism; the scheduler above does. A worker the pool replaces during a run (a cancelled task outlived its grace) gets a new port; the genes on their way to the old one are read again.
Two groups, read in their cells. Differential expression of two groups streams through the same loop over the groups’ cells only (runGeneStream’s cells): each read is a block of genes in those cells (DataLayer.deliverGeneCellSubset over the route, readGeneCellSubset without one; data/gene-delivery.js block), which the reader reads by cell or by gene, whichever costs less. A read by cell decodes every row it touches whatever its genes, so a read takes as many tasks as the memory gate has room for (bufferedTasks): every gene in one read when they fit, else several reads, each within the gate, never every gene at once. Until a read is measured a gene of a block is taken to hold a position and a value a cell (CELL_SUBSET_ENTRY_BYTES).
Counted while held. What a run has read and not yet computed (being read, delivered and waiting, in a task) is a holder of the analysis share while the run goes (GENES_IN_FLIGHT_LABEL, given back however the run ends), so memoryReport() shows it and the caches of the share make room for it. An analysis that computes on the page over many genes (signature scores) reads them through visitGeneColumnsHere: a batch at a time within the same gate, each gene visited and let go, the batch counted the same way; the scorers’ per-cell state is counted too while it is built.
Cancellation. A finished or aborted run cancels its reads still waiting in the scheduler (they never start) and those under way; a shared read is cancelled only when every caller has left. Each dataset load has its own scheduler: replacing the dataset closes the previous one, which aborts and rejects every read it still held, so a stalled read of the old dataset never holds back the new dataset’s reads. No task has a deadline: a task is one synchronous call with no progress to measure, so a bound would only turn a slow success (10M cells on few cores) into a failure of every window’s tasks. A task stops through its signal, and a worker that does not come back from a cancelled task is replaced. A failed task’s error reaches the caller with its class (utils/error-wire.js).
Progress. Every read is reported to the read activity log (data/read-activity-log.js): a gene the user waits for shows in its loading notification, and a long read is written to the console as [cellucid:read].

4) Plot infrastructure (Plotly + custom helpers)#

Plots are managed via:

a plot registry / factory (analysis/plots/plot-factory.js)
a Plotly loader (analysis/plots/plotly-loader.js) to keep initial load light: Plotly is not part of startup; opening an analysis (_setAnalysisMode) starts loading it at the page’s next idle moment (prefetchPlotly), so the library’s evaluation, one long main-thread task, happens while the analysis is being set up rather than inside the run that draws the first plot
plot types registered as modules under analysis/plots/types/

Theme integration:

CSS variables must be resolved into concrete colors for Plotly.
Centralized in analysis/shared/plot-theme.js and updated on cellucid:theme-change.

5) Memory and cleanup#

What the analyses keep by choice is counted by the app’s one memory model (data/memory-budget.js, Memory budgets), each where it is held: the page values the data layer keeps (analysisPages), kept marker runs and the per-cell state runs keep in the analysis workers (analysis, every worker’s copy counted). The model trims them, oldest first, when the allowance or the working set changes, a refused allocation lowers the allowance, or the page is hidden; nothing in flight is invalidated and no window is closed. The analysis pool runs no more workers than the analysis share holds a run’s per-cell state for, and its workers end once idle for 10 s (workers/idle-retirement.js), starting again for the next task.

Components still clean up what they own (plots purged, caches cleared on a dataset reload): cellucid/assets/js/app/analysis/shared/resource-cleanup.js.

Actions, history and sessions (analysis windows + results)#

Every analysis change is an action (app/actions/definitions/analysis.js, provided by app/analysis/actions.js installAnalysisActions once the module exists): analysis.open, analysis.close, analysis.pages.set, analysis.run, one analysis.<mode>.set per mode with options (its parameters made from the mode’s options schema), and the window actions analysis.window.copy, analysis.window.close, analysis.window.place, analysis.window.collapse and an expanded view’s analysis.size.set; the reads analysis.list, analysis.result, analysis.<mode>.get and analysis.cancel answer without a step and make no state: the panel makes a mode’s analysis on first use, and a mode it made is part of the analysis domain, so a read of a mode the panel never made goes through ComparisonModule.readPanelUI, which makes it for a synchronous read and unmakes it in the same task (a read that answers a promise is refused), so the panel is as it was when the read ends. ComparisonModule.getUI only peeks (AnalysisUIManager.peekUI) and never makes one; one analysis reading another’s results, as the dot plot and the gene signature read the marker genes, peeks the same way. A person’s changes in a panel or a window are recorded by observation (recordObserved), so each is one history step.

What the analyses hold is the analysis state domain (app/state/domains/analysis.js): the floating windows (mode, title, settings, result) and the docked panel’s open mode with every opened mode’s settings and result. A result is kept as the analysis exported it (analysis/shared/analysis-result-session.js: JSON and typed arrays, the arrays by reference) and put back as it was, never computed again by a restore. A share that leaves results out writes each as the record of a result left out (leftOutResultOf: its kind and what it was computed on, no payload); once that session file or saved state is shown, session/recompute-results.js runs each analysis at the head whose result was left out again through analysis.run (origin session, an ordinary step), compares what the new result was computed on with the shared one (result-provenance.js staleReasons) and says why where they differ. Only opening does this, never a jump; an analysis that is not shown is refused (it cannot draw) and keeps its record, so the next opening tries again. A restore reopens the windows from their descriptors (createFromSessionDescriptor) and hands the docked panel its record (importSessionState) only when they differ. Where each window stands, its size, whether it is folded and the sizes remembered per analysis are page layout: the layout domain holds them (layout.analysis, read and put back through exportWindowPlaces / applyWindowPlaces and exportViewSizes / importViewSizes), so the window actions that only move, resize or fold a window touch layout, and opening, copying or closing one touches both. A session file holds the domain like every other (Action layer, state domains and history, Sessions: persistence and serialization).

Performance and correctness edge cases#

1) Variable selection + mutual exclusion#

Many analysis UIs treat variable selection as mutually exclusive:

categorical obs vs continuous obs vs gene expression

When adding new modes, be explicit about:

which variable types are accepted
what happens when the selection becomes invalid mid-session (field deleted, dataset switched)

2) Visibility vs highlight membership#

Common pitfall:

“highlight group contains 10k cells” but filters hide 9k; analysis must decide which notion it uses.

Recommendation:

expose the choice in UI (visible-only vs all highlighted) when it affects interpretation.

3) Dataset reload#

On dataset reload (especially “in-place” reloads for local-user), analysis must:

clear caches tied to the old dataset
purge old plots and detach event handlers
reset page selection safely if page ids changed

To inspect it in the console, use window._comparisonModule.

4) The per-gene streaming path#

Differential expression and marker discovery repeat the same steps once per gene, so anything in that loop is multiplied by the gene count. The loop keeps these properties, and each one is pinned by a test:

Reading, decoding and the genes’ values stay off the main thread. A gene’s values are read by the AnnData reader worker (data/anndata-reader-worker.js), whose decode workers decompress the chunks, and delivered from there to the analysis worker that computes them over the run’s gene route: the page sends which genes each task takes, hears their bytes and takes in each task’s statistics, packed (one message a task), and never holds a gene a run computes (see Reading AnnData directly).
A chunk goes to a decode worker with room (formats/io/decode-pool.js): queued chunks are sent as workers free up, a gene index being built in the background waits behind every other read, small chunks waiting together travel as one message, and a small chunk alone is decoded in the reader thread. The workers start when a file opens; there are as many as the cores (less the page and the reader) and their memory budget allows, the budget as it is when one may start; idle ones end after 10 s or when the memory model trims their share.
Per-run work is done once per run. A run’s groups reach each worker once (a context the pool replays to a replaced worker), and a gene’s task carries only its delivery; the per-gene work on the main thread is placing each task’s packed results.
Rank tests tabulate values without a Map. Wilcoxon (DE and markers), Spearman ranks and order statistics number distinct values through DistinctValueIndex in compute/math-utils.js up to 4,096 distinct values and sort natively above that. Results are bit identical to the sort-based definition (tests/analysis-engine-rank-kernel-identity.test.mjs, tests/statistical-tests.test.mjs).
Page membership is recomputed only when the page changes. DataLayer keeps each page’s membership and digest beside the structure it came from (the page, its group list, each group, its enabled flag, its cellIndices array and length) and reuses them while all of those are the same objects. Highlight cellIndices arrays are therefore never edited in place once published; replace the array instead.
Yields are time-sliced. Long loops hand the event loop back every 12 ms through a posted message (createSlicedYield in shared/concurrency-utils.js), not after a fixed item count through setTimeout(0), whose 4 ms clamp turned a 20,000-gene run into seconds of idle waiting.
Fold changes past float64. “Automatic” on a file that records uns['log1p'] takes scanpy’s formula log2((expm1(a) + 1e-9) / (expm1(b) + 1e-9)). expm1 of a mean above about 709.78 is infinite in double precision, so a file whose values are not logged although it records uns['log1p'] (EBI SCEA E-ANND-5: per-million values of 252–1,118 under an empty uns['log1p']) gave ±Infinity or NaN, and differential expression failed to draw. Where the ratio or a term overflows, foldChangeOfMeans (compute/gene-statistics.js) takes the same formula in log space (logExpm1PlusEpsilon: ln(expm1(y) + ε) = y + ln1p((ε − 1)e^−y)), so DE, marker genes and pseudobulk get its finite value; below the overflow the value is scanpy’s bit for bit (tests analysis-fold-change-scaled-data, the scanpy parity fixture). On E-ANND-5, cells 0–399 against 400–799 over 33,538 genes: 32 infinite or NaN fold changes before, none after. Such means also show the matrix is not log1p: the user guide’s DE page says to set Values to As stored under Data mapping. Python and R compute no DE.

Troubleshooting (analysis)#

Symptom: “Analysis panel is blank”#

Likely causes (ordered):

#page-analysis-section DOM element missing (HTML change).
Analysis module threw during init (console error).
Plotly failed to load (blocked network or CSP).

How to confirm:

Console: search for the first error after [Main] logs.
Check whether window._comparisonModule exists.

Fix:

Restore the expected DOM container id.
For Plotly issues: confirm analysis/plots/plotly-loader.js can fetch its assets under your CSP.

Symptom: “Plots get slower over time / tab memory grows”#

Likely causes:

caches not being cleared on dataset reload
Plotly plots not being purged (old DOM nodes retained)
missing cleanup handler registration