Round trip (results back from the viewer, columns and gene sets into it)#

What a person makes in Cellucid – highlight groups (lasso, k-NN, proximity, category and range picks), labels and annotations, clusters made from pages, gene-signature scores, gene sets – comes back into your AnnData (or MuData) in one call, and what you compute in Python goes into the viewer the same way. R’s cellucid_pull(), cellucid_push() and friends do the same for Seurat, SingleCellExperiment and data frames (Public Functions).

You want…	Call…
everything made in a live viewer, as `obs` columns and gene sets	`AnnDataViewer.pull()` (or `pull()` with the viewer)
the same from the web app’s exports, without a viewer	`pull()` with the exported files or their folder
the cells of a highlight group or the last selection, as an AnnData	`AnnDataViewer.subset()`
the last selection, with obs names	`AnnDataViewer.selection`
every gene set the viewer holds	`AnnDataViewer.gene_sets()`
a column (labels or a score) shown in the viewer	`AnnDataViewer.push()`
a gene set offered in the viewer	`viewer.gene_set_save(name=..., genes=...)`
a GMT file	`write_gmt()` / `read_gmt()`
cells highlighted from Python, by row, mask or obs name	`viewer.highlight_cells(cells=...)`

Nothing waits#

The viewer sends what is made as it is made: once its dataset opens (and whenever its channel opens again) every result and then an inventory, and each change after. AnnDataViewer.results is that copy, and pull, subset, gene_sets and selection read it at once, with no request. In a notebook the widget hears the viewer between cells, so a lasso drawn and viewer.pull(adata) run in the next cell finds it; a browser-tab AnnDataViewer works the same way.

One page, one dataset. The copy is of one epoch: one page’s account of one dataset it opened. A reloaded page, or the viewer shown again in another output, replaces what an older page sent, whatever its revisions. While the person has another dataset open in the viewer (one picked in it instead of the notebook’s), results and everything reading it raise, naming the dataset to open again: that dataset’s cells are not adata’s.
Pushes and mappings wait for the viewer, not the cell. push, gene_set_save and mapping_set (like every action) send at once and return their request. A column of a format only the browser reads goes once the viewer has said how many cells it has – so a notebook run top to bottom (“Run All”) works. The field or set is listed in viewer.results, and the mapping is AnnDataViewer.data_mapping, from a later cell; anything refused fails its request (request.result() raises with the reason) and is printed unless a command_error hook handles it.
Sessions in two steps. A session (the viewer’s history) is asked for, then collected – the one way, in a notebook (in a later cell) and a browser tab alike: request = viewer.request_session(), then request.result() or viewer.apply_session_to_anndata(adata, request), which returns the plan (plan.apply() writes it) (R: cellucid_request_session(), then cellucid_session_result() or cellucid_apply_session(object, request), then cellucid_apply_plan()).

Fast path#

import cellucid

viewer = cellucid.show_anndata(adata, dataset_name="My study", dataset_id="my-study-v1")
# … lasso cells, label clusters, score a gene signature in the viewer, then:
summary = viewer.pull(adata)
summary.columns                       # every obs column written, and what it holds
adata.uns["cellucid"]["gene_sets"]    # every gene set, by name

viewer.push(adata.obs["leiden"])      # the clusters, as a field in the viewer
viewer.gene_set_save(name="T cells", genes=["CD3E", "CD3D", "IL7R"])
t_cells = viewer.subset(adata, "Lasso selection")

Without a notebook, export from the web app (Highlights: CSV or .h5ad; a field’s Export button; a gene set editor’s Export; Save session), then:

cellucid.pull(adata, "~/Downloads/cellucid-exports")   # a folder, a file, or a list of files

What `pull` writes#

Result	`obs` column	dtype
a highlight group	`cellucid_highlight__<group id>` (`highlight_prefix`)	`bool`
a user-defined categorical field (labels, annotations, clusters)	`<key>` (`field_prefix`)	`category`, the viewer’s categories in order
a continuous field, a score (a gene signature’s: `<set> score`)	`<key>` (`field_prefix`)	`float32`
the cells the filters leave visible (`visibility=True`)	`cellucid_visible`	`bool`
a gene set	–	`uns["cellucid"]["gene_sets"][name]`: `genes`, `symbols` (from a viewer whose Data mapping names the genes by a var column), `description`, `origin`, `pulled_from`, `id`

Genes by identifier. A gene set’s genes are the var index, as adata.var_names holds them. When the viewer’s Data mapping names the genes by a var column (such as a CELLxGENE dataset’s feature_name), symbols holds the names it shows for them; named by the var index, the identifiers are the names and there is no symbols. AnnDataViewer.gene_sets() returns the identifiers, or with symbols=True the names shown. A gene set name used twice is "name (2)", "name (3)"…, as everywhere in Cellucid.
Missing values. A cell with no category or no value is missing (NaN).
Subsamples. When the viewer opened a subsample, every result names cells by their row in obs (original positions). A cell the viewer did not open is <NA> in a highlight column (the nullable boolean dtype) and missing in a field’s: unknown, not “not selected”.
The cells in the viewer’s order. A viewer names a cell by its original position: an object’s own row, or, for a file prepare() wrote (cells stored in priority order, uns["cellucid"]["priority"] verified against its cells), the original position its record gives the row. pull and subset from a viewer refuse an object holding the viewer’s cells in another order – that file read with anndata.read_zarr as stored (put it back with restore_original_order()), or an object sorted since – and an object whose cells the viewer’s dataset does not have; nothing is written. R’s cellucid_pull() and cellucid_subset() refuse the same.
Again and again. uns["cellucid"]["pulled"] records every column a pull wrote: where it came from (pulled_from: "viewer", or the file’s name) and the result it holds (kind, id). A pull of the same result writes the same column again, whatever suffix it took. With prune=True (the default) a viewer pull removes the columns – and the gene sets – of an earlier viewer pull whose result the viewer no longer holds. A result left out of a pull (highlights=False, a pushed field without include_pushed) keeps its column; a file’s columns are never removed.
Conflicts. A column of the same name that no pull from the same source wrote is a conflict: on_conflict="error" (the default) raises and writes nothing, "replace" overwrites it, "suffix" writes <name>_2 beside it (and the next pull writes <name>_2 again). Session files follow the same rule.
Pushed columns. A field this notebook pushed comes back only with include_pushed=True: adata holds it already.
One call, one write. Every column is built first and obs changes once: a pull that fails leaves adata as it was. A MuData takes the columns on its global obs.

apply_cellucid_session_to_anndata() plans, then writes, the state at any step of a saved session (its highlight groups and fields, with the same column rules, its gene sets and its renames). pull of a .cellucid-session file writes the highlight groups and user-defined fields of the state the session shows (its active tracker’s head), checked against adata as an apply checks it (the dataset, the cells and their order, the fields that state uses), names them with every other column of the pull, and adds the session’s gene sets; the renames are the apply’s to write.

The web app’s exports#

Export	File	Read back as
Highlights › CSV	a `highlight` column `cellucid_highlight__<group id>` per group of the page (`true`/`false`), a row per open cell	one `bool` column per group
a field’s Export	the field under its name, of kind `categorical` or `continuous`, a row per open cell	a `category` or `float` column
Highlights › .h5ad	the highlighted cells as an AnnData, in the dataset’s own order (ascending original position): obs names, every obs field (continuous ones as `float32` or `float64`, as the viewer holds them), each group as a `bool` column, the genes by identifier (the var index; the shown names in `var['symbol']` when they differ), the expression rows (CSR, `float32`) of the matrix the data mapping reads, the embeddings shown (`float32` or `float64`, as stored). A key an HDF5 file cannot name – one holding `/`, or `.` – is written renamed (`/` as `_`), and the app says which	`anndata.read_h5ad`
a gene set editor’s Export, Marker genes › GMT	GMT, genes by identifier	`read_gmt()`, `uns["cellucid"]["gene_sets"]`
Save session	`.cellucid-session`	`apply_cellucid_session_to_anndata()`

An obs-column CSV declares itself, so nothing is guessed from its values:

obs_name,leiden,cellucid_highlight__highlight_1,score
kind,categorical,highlight,continuous
AAACCTG-1,0,true,0.25
AAACGGG-1,10,false,

The header row names the columns (obs_name first); the kind row (kind first) gives each column’s kind: highlight (true, false, or empty for unknown), categorical (labels – 0, 1, 10 or True stay labels; empty: no category) or continuous (numbers; empty: no value). A CSV without a kind row is refused, saying how to add one. A cell the file does not list is unknown (<NA>, or missing), as an unopened cell of a subsample; a cell the file names that adata does not have is refused, naming it.

GMT files follow one rule in the web app, Python and R: cells are tab-separated and trimmed; each run of tabs and line breaks inside a name, description or gene is written as one space; a line without a name reads as Set <n>; a name used again is name (2), name (3)…; each gene is kept once. write_gmt() writes a name that repeats an earlier one, once cleaned, as read_gmt() will name it.

Pushing into the viewer#

AnnDataViewer.push() takes a pandas.Series aligned by its index (obs names: every cell, or some, the rest without a value) or an array of one value per cell. Categorical, string, object and bool values become a categorical field (a bool column’s labels are "false" and "true", as R pushes a logical one); numbers a continuous one. Pushing the same key again replaces it; a field of that key the notebook did not push is never replaced (the viewer refuses it, and the request’s result() raises the reason). A pushed column is saved with the session (a step of its history, the column’s values in that step’s state).

viewer.gene_set_save(name=..., genes=...) (the gene-set.save action) adds a gene set (or replaces the one of that name), its genes by identifier (adata.var_names) or by the names the viewer shows when a var column names them (identifiers first): Gene signature and Dot plot offer it under From notebook, and it is saved with the session (its gene-sets state).

The wire (for developers)#

Results travel on the viewer channel (Architecture: the viewer channel, protocol schema 4, viewer-protocol.json):

the result event (binary): a header {type, epoch, revision, kind, id, nObs, meta, cells, values} and a body of cells original positions (uint32, strictly ascending, none when cells is null) then values (one per named cell, or per cell of the dataset when cells is null; a code type’s largest value is “no category”, NaN “no value”). Kinds: highlight, categorical, continuous, geneSet (genes by identifier, the var index, and symbols, the names shown in the same order when a var column names the genes, else null), visibility (visible), subset (open, the open cells);
result_removed {epoch, revision, kind, id} and results {epoch, dataset, revision, nObs, items}, the inventory ending a snapshot: after a dataset opens (a page reload included) and whenever the channel opens again the viewer sends every result and the inventory, so a notebook’s copy is replaced whole. dataset is the id of the notebook’s dataset the results describe, or null while another one is open in the viewer (then items is empty and no result, selection, hover or click is sent);
an epoch (16 hexadecimal characters, drawn at random) is one page’s account of one dataset: a new page, and every dataset a page opens, start one, with revisions from 1. The notebook keeps one epoch’s results; an event of another epoch starts afresh. Within an epoch revisions grow with every result and removal: a frame whose revision the notebook already holds is one it applied;
a pushed column (field.push, a binary action) and a gene set (gene-set.save) are actions: each is answered by its action_result (the outcome names the field or set it made; the result it made follows as for any result) or by a command_error with its requestId;
color_by {field, source, symbol, cells_read}: a gene by its identifier, and the name the viewer shows for it when a var column names the genes (symbol; null otherwise, and for an obs field); for a gene read for a sample of the open cells (a large dataset stored cell by cell opens every cell on the map and reads a gene for the first of them: Genes on a sample of the cells), how many of the open cells it is read for (cells_read; null when every open cell’s are, and for an obs field), sent again when that grows. AnnDataViewer.results is the live copy; viewer.register_hook("result", …) runs on each. Nothing in the round trip is a request the notebook waits on.

API reference#

cellucid.pull(adata, source, *, highlights=True, fields=True, scores=True, visibility=False, gene_sets=True, include_pushed=False, highlight_prefix='cellucid_highlight__', field_prefix='', on_conflict='error', prune=True)[source]#

Write what was made in Cellucid onto adata in one call, in place.

Every highlight group (a bool column each), every user-defined categorical field – labels, annotations, clusters – (category), every continuous field and score (float32) go to adata.obs (a MuData’s global obs), and every gene set to adata.uns['cellucid']['gene_sets']: its genes by identifier – the var index, adata.var_names – and, from a viewer whose data mapping names the genes by a var column (such as feature_name), the names it shows for them (symbols).

Parameters:

adata (anndata.AnnData or mudata.MuData) – The object the viewer shows, with its cells in the same order. From a viewer, an object holding the viewer’s cells in another order is refused (nothing is written), as is one read from a Cellucid-ready file in its stored order (obs['cellucid_original_position']; cellucid.restore_original_order() puts it back).
source (AnnDataViewer, path, or list of paths) – A viewer (a notebook widget or a browser-tab viewer): what it holds, as it last sent it – every result as it was made, and all of them whenever it opened its dataset, so a cell run after the work pulls it all, with no waiting – or what the web app exports with no notebook: obs-column .csv files (a header row, obs_name then each column’s name; a kind row, kind then each column’s kind, highlight, categorical or continuous; a row per cell; each column written as named), .gmt gene sets and .cellucid-session files (the state the session shows, its active tracker’s head: its highlight groups, user-defined fields and gene sets, checked against adata as an apply checks it; apply_cellucid_session_to_anndata() applies another step, and the session’s renames) – one file, a list, or a folder of them.
highlights (bool) – Which results to write: highlight groups; user-defined fields; scores an analysis made (a gene signature’s); the filter’s visible cells (cellucid_visible, off by default); gene sets.
fields (bool) – Which results to write: highlight groups; user-defined fields; scores an analysis made (a gene signature’s); the filter’s visible cells (cellucid_visible, off by default); gene sets.
scores (bool) – Which results to write: highlight groups; user-defined fields; scores an analysis made (a gene signature’s); the filter’s visible cells (cellucid_visible, off by default); gene sets.
visibility (bool) – Which results to write: highlight groups; user-defined fields; scores an analysis made (a gene signature’s); the filter’s visible cells (cellucid_visible, off by default); gene sets.
gene_sets (bool) – Which results to write: highlight groups; user-defined fields; scores an analysis made (a gene signature’s); the filter’s visible cells (cellucid_visible, off by default); gene sets.
include_pushed (bool) – Also write the fields this notebook pushed (viewer.push), which adata holds already.
highlight_prefix (str) – Each highlight’s column is <highlight_prefix><group id>; each field’s and score’s <field_prefix><key>.
field_prefix (str) – Each highlight’s column is <highlight_prefix><group id>; each field’s and score’s <field_prefix><key>.
on_conflict ({"error", "replace", "suffix"}) – What a column name that adata.obs has already does, when no pull from the same source (the viewer, or the same file) wrote it: raise (nothing is written), overwrite it, or write <name>_2 beside it. A result an earlier pull wrote is written to the same column again, suffix and all.
prune (bool) – For a viewer: remove the columns, and the gene sets, an earlier viewer pull wrote whose result the viewer no longer holds (a highlight deleted since). A result left out of this pull (highlights=False, a pushed field) keeps its column, and files never remove anything.

Returns:

The columns written (columns), removed (removed), the gene sets brought (gene_sets) and removed (removed_gene_sets), and the results left out (skipped).

Return type:

PullSummary

Notes

When the viewer opened a subsample, a cell it did not open is <NA> in a highlight column and missing in a field’s: unknown, not “not selected”; a cell an obs-column CSV does not list is the same. adata.uns['cellucid']['pulled'] records each column written, with where it came from (pulled_from) and the result it holds (kind, id); each gene set in uns['cellucid']['gene_sets'] the same.

cellucid.write_gmt(gene_sets, path, *, descriptions=None)[source]#

Write gene sets as a GMT file (MSigDB): one set per line, name<TAB>description<TAB>gene....

Parameters:

gene_sets (Mapping[str, Sequence[str]]) – Each set’s name and its genes, in order.
path (str | PathLike[str]) – The file to write.
descriptions (Mapping[str, str] | None) – A description per set name; empty when not given.

Returns:

The file written. Each run of tabs and line breaks in a name, description or gene is written as one space; a name that then reads as an earlier one’s is written "name (2)" (as read_gmt() names a repeated name); each gene once. The Cellucid web app reads it (Gene signature and Dot plot: Import), as do read_gmt() and cellucid-r’s cellucid_read_gmt().

Return type:

Path

cellucid.read_gmt(path)[source]#

Read a GMT file into {set name: [genes]}, in file order.

One rule with the web app and cellucid-r: cells are tab-separated and trimmed; blank lines are skipped; a line without a name is "Set <n>" (its place among the sets); a name used again is "name (2)", "name (3)"…; each gene is kept once.

Return type:: dict[str, list[str]]
Parameters:: path (str | PathLike[str])

class cellucid.PullSummary(columns=<factory>, removed=<factory>, gene_sets=<factory>, removed_gene_sets=<factory>, skipped=<factory>)[source]#

Bases: object

What pull() wrote.

columns maps each obs column written to what it holds (kind, id, pulled_from and the result’s label, key or page); removed lists the columns of an earlier viewer pull whose result the viewer no longer holds; gene_sets maps each gene set brought to its genes, and removed_gene_sets names the gene sets of an earlier viewer pull the viewer no longer holds; skipped names results left out and why.

Parameters:

columns (dict[str, dict[str, Any]])
removed (list[str])
gene_sets (dict[str, list[str]])
removed_gene_sets (list[str])
skipped (dict[str, str])

columns: dict[str, dict[str, Any]]#

removed: list[str]#

gene_sets: dict[str, list[str]]#

removed_gene_sets: list[str]#

skipped: dict[str, str]#

__init__(columns=<factory>, removed=<factory>, gene_sets=<factory>, removed_gene_sets=<factory>, skipped=<factory>)#

Parameters:

columns (dict[str, dict[str, Any]])
removed (list[str])
gene_sets (dict[str, list[str]])
removed_gene_sets (list[str])
skipped (dict[str, str])

Return type:

None

class cellucid.Selection(cells, obs_names, source)[source]#

Bases: object

The cells of the viewer’s last selection: original positions, their obs names (when the viewer’s dataset has them in this process), and the tool that made it.

Parameters:

cells (ndarray)
obs_names (Index | None)
source (str)

cells: ndarray#

obs_names: Index | None#

source: str#

__init__(cells, obs_names, source)#

Parameters:

cells (ndarray)
obs_names (Index | None)
source (str)

Return type:

None

class cellucid.ViewerResult(kind, id, revision, n_obs, meta, cells, values, column=None, source='viewer', known=None)[source]#

Bases: object

One result the viewer holds (the protocol’s result).

cells are original positions (rows of obs), ascending, or None for every cell; values are a field’s codes or numbers, one per cell named (or per cell of the dataset), or None.

Parameters:

kind (str)
id (str)
revision (int)
n_obs (int)
meta (dict[str, Any])
cells (ndarray | None)
values (ndarray | None)
column (str | None)
source (str)
known (ndarray | None)

kind: str#

id: str#

revision: int#

n_obs: int#

meta: dict[str, Any]#

cells: ndarray | None#

values: ndarray | None#

column: str | None = None#: The obs column it is written to, when it names one itself (a column of an obs-column CSV); None follows the naming scheme.

source: str = 'viewer'#

"viewer", or the name of the file it came from.

Type:: Where it was read

known: ndarray | None = None#: A highlight’s known cells, when the source says which (an obs-column CSV lists them); None: the cells the viewer opened.

classmethod from_event(payload)[source]#

The result of one decoded result event payload.

Return type:: ViewerResult
Parameters:: payload (Mapping[str, Any])

__init__(kind, id, revision, n_obs, meta, cells, values, column=None, source='viewer', known=None)#

Parameters:

kind (str)
id (str)
revision (int)
n_obs (int)
meta (dict[str, Any])
cells (ndarray | None)
values (ndarray | None)
column (str | None)
source (str)
known (ndarray | None)

Return type:

None

The viewer methods (AnnDataViewer.pull(), AnnDataViewer.results, AnnDataViewer.selection, AnnDataViewer.obs_names, AnnDataViewer.subset(), AnnDataViewer.gene_sets(), AnnDataViewer.push(), and the action methods such as gene_set_save and highlight_cells) are documented with the class in Viewer classes (Jupyter objects you control from Python).

Round trip (results back from the viewer, columns and gene sets into it)#

Nothing waits#

Fast path#

What pull writes#

The web app’s exports#

Pushing into the viewer#

The wire (for developers)#

API reference#

What `pull` writes#