Data preparation (`prepare`)#

This page documents prepare(), which writes a Cellucid-ready AnnData, and restore_original_order(), which reads one back in its cells’ original order. A Cellucid-ready AnnData is one standard AnnData (.h5ad, .zarr folder, or .zarr.zip) laid out for the web app’s range reads and naming itself in uns['cellucid']. Given a MuData it writes a Cellucid-ready MuData (.h5mu, or MuData Zarr as a folder or .zarr.zip): each modality a Cellucid-ready AnnData. The result can be:

opened in the web app (Open…, or the Link or server bar once hosted),
served with cellucid serve <file>,
embedded in notebooks with show_anndata(),
read by any AnnData tool (anndata.read_h5ad, anndata.read_zarr).

Long-form walkthroughs: Preparing an AnnData for Cellucid. The written contract: Cellucid-ready AnnData specification.

Fast path (copy/paste)#

from cellucid import prepare

# Everything the AnnData holds (obs, X with var, obsm, obsp['connectivities']):
path = prepare("my_study.h5ad", format="h5ad", data=adata, dataset_id="my-study-v1", dataset_name="My study")

# The same, component by component; an argument given with data= replaces that component.
path = prepare(
    "my_study.h5ad",
    format="h5ad",
    force=True,
    embeddings={"X_umap": adata.obsm["X_umap"], "X_pca": adata.obsm["X_pca"]},
    dataset_id="my-study-v1",
    dataset_name="My study",
    obs=adata.obs,
    var=adata.var,
    gene_expression=adata.X,
    connectivities=adata.obsp["connectivities"],
    latent_key="X_pca",
)

prepare() returns the pathlib.Path it wrote.

Signature#

prepare(
    out_path,
    *,
    format,
    data=None,
    embeddings=None,
    dataset_id=None,
    dataset_name=None,
    obs=None,
    var=None,
    gene_expression=None,
    connectivities=None,
    vector_fields=None,
    modalities=None,
    gene_identifiers=None,
    dataset_description=None,
    citation_text=None,
    citation_url=None,
    source_name=None,
    source_url=None,
    embedding_defaults=None,
    latent_key=None,
    obs_keys=None,
    var_gene_id_column=None,
    vector_field_default=None,
    normalize_embeddings=None,
    centroid_min_points=None,
    modality=None,
    priority_seed=0,
    zarr_format=None,
    force=False,
) -> Path

Every argument after out_path is keyword-only. format is required and has no default. embeddings, dataset_id and dataset_name are required unless data holds them (obsm, uns['cellucid']['id'], ['name']). The R package’s cellucid_prepare() takes the same arguments and writes the same Zarr layout (format = "zarr" or "zarr.zip"; R writes no .h5ad or .h5mu, so a MuData from R is always MuData Zarr).

Which `format`?#

`format`	Writes	Choose it for
`"zarr"`	a `.zarr` folder of small files	GitHub: no file over 100 MB, no LFS, many small files at once over HTTP/2
`"h5ad"`	one `.h5ad` file	everything else: single-file hosts (Zenodo, Figshare, Hugging Face, S3, a lab server), local files, `cellucid serve` and Jupyter
`"zarr.zip"`	the Zarr store in one `.zarr.zip` file	wherever an `.h5ad` goes, as Zarr
`"h5mu"`	one `.h5mu` file (a MuData only)	a MuData wherever an `.h5ad` goes; `"zarr"` and `"zarr.zip"` write a MuData as MuData Zarr

The measurements behind this are in Which format?.

out_path must end in the suffix of format; a mismatch is refused with the path to write instead.

Argument	Stored as
`out_path`	the output; ends in `.h5ad`, `.zarr`, `.zarr.zip` or `.h5mu`, matching `format`
`format`	the container: `"h5ad"`, `"zarr"`, `"zarr.zip"` or `"h5mu"` (required)
`data`	an AnnData or a MuData whose components are the defaults of the arguments below (Prepare an AnnData or a MuData whole (data=))
`embeddings`	`obsm[key]`, float32, every column
`dataset_id`, `dataset_name`	`uns['cellucid']['id']`, `['name']`; `uns['title']`
`obs`, `var`	`obs`, `var` (repeated text as categorical)
`gene_expression`	`X`: CSC when sparse, float32 (float64 for counts above 2^24)
`connectivities`	`obsp['connectivities']`, symmetric CSR float32
`vector_fields`	`obsm['<field>_<basis>']` for each embedding `X_<basis>` (`velocity_umap`)
`modalities`	with a MuData, the modalities written, in order: several as a MuData of those, one as an AnnData
`gene_identifiers`	keeps those genes, in that order
`dataset_description`, `citation_text`/`citation_url`, `source_name`/`source_url`	`uns['cellucid']['description']`, `['citation']`, `['source']`
`embedding_defaults` … `centroid_min_points`, `modality`	`uns['cellucid']['defaults']` (only the ones given); `modality` (a MuData’s) is the modality the viewer opens on
`latent_key`	also `uns['cellucid']['outlierQuantiles']`: every categorical `obs` column’s outlier quantiles in that latent space
`priority_seed`	the cells’ order: by ascending priority, the subsampling `uniform` rank under this seed (default 0), recorded in `uns['cellucid']['priority']`, each row’s original position in `obs['cellucid_original_position']`, each gene’s values below each prefix level in `varm['cellucid_prefix_counts']`; `None` stores the cells as given (Priority order)
`zarr_format`	2 (default) or 3, for `format="zarr"` or `"zarr.zip"`; refused with `"h5ad"` and `"h5mu"`
`force`	replace an existing output of the same kind

Interface reference#

The whole Cellucid window with the Pancreas sample coloured by cell_type: the sidebar shows Coloring and filtering with the categorical field selected and its eight-row legend, and the 3D embedding fills the canvas in the matching colours. — A loaded dataset in Cellucid: the sidebar controls the active view while the categorical legend maps directly to the colored points.#

API reference#

cellucid.prepare(out_path, *, format, data=None, embeddings=None, dataset_id=None, dataset_name=None, obs=None, var=None, gene_expression=None, connectivities=None, vector_fields=None, modalities=None, gene_identifiers=None, dataset_description=None, citation_text=None, citation_url=None, source_name=None, source_url=None, embedding_defaults=None, latent_key=None, obs_keys=None, var_gene_id_column=None, vector_field_default=None, normalize_embeddings=None, centroid_min_points=None, modality=None, priority_seed=0, zarr_format=None, force=False)[source]#

Write a Cellucid-ready AnnData (or MuData): a standard file laid out for the browser.

The output is an ordinary AnnData that anndata.read_h5ad or anndata.read_zarr reads, written so the Cellucid web app reads it fast over HTTP: the expression matrix gene-major (CSC), arrays chunked for range requests and compressed with codecs browsers decode natively, and the dataset’s name, citation and launch defaults in uns['cellucid']. Open it with cellucid serve, show_anndata(), or the web app directly. The written contract is the “Cellucid-ready AnnData” page of the documentation; cellucid-r’s cellucid_prepare() writes the same.

The inputs come from data – an AnnData or a MuData – or are passed one by one, or both: every component data holds is its default, and an argument given explicitly replaces that one component. The output is byte for byte what the same components passed by hand write.

A MuData is written as a Cellucid-ready MuData (format="h5mu", "zarr" or "zarr.zip"): a standard MuData whose modalities share the cells, each modality a Cellucid-ready AnnData (gene-major X, chunked and compressed for the browser), the cells’ metadata, shared embeddings and graph at the root with uns['cellucid']. The written contract is the “Cellucid-ready MuData” section of the specification.

Every input is checked before anything is written (a MuData modality’s matrix as it is written, one modality at a time), and the output is written beside out_path under a temporary name and moved into place only once it is complete.

Parameters:

out_path (str | PathLike[str]) – Where to write, ending in the suffix of format: .h5ad, .zarr, .zarr.zip or .h5mu. A leading ~ is expanded.
format (Literal['h5ad', 'zarr', 'zarr.zip', 'h5mu']) –
Required, with no default – which one depends on where the data goes ("h5mu" is the single-file form of a MuData, like "h5ad"; "zarr" and "zarr.zip" write a MuData as MuData Zarr):
- "zarr": a .zarr folder of small files, for GitHub (a repository or its Pages site). GitHub refuses files over 100 MB and answers over HTTP/2, 100 requests at a time, so a folder’s one request per chunk costs nothing there.
- "h5ad": one .h5ad file, for everything else: single-file hosts (Zenodo, Figshare, Hugging Face, S3, a lab server), local files, cellucid serve and Jupyter. Its ranges are merged into few requests; most such hosts answer over HTTP/1.1, six requests at a time, where a folder is several times slower. What opening reads comes first in the file (431 KB of a 561,947-cell atlas of 5,103 genes), which the web app reads with its first request of 512 KiB.
- "zarr.zip": the Zarr store in one .zarr.zip file (entries stored, metadata last), wherever an .h5ad goes.
Measured on a 561,947-cell atlas, every read the web app makes, in total: over GitHub’s HTTP/2 (80 ms, 50 MB/s) 8.3 s from the .h5ad, 8.0 s from the folder, 8.2 s from the .zarr.zip; over HTTP/1.1 (80 ms, 50 MB/s) 9.0 s, 46.8 s and 9.0 s; from a local file in a browser 2.5, 2.6 and 2.7 s. The .zarr.zip is 13% smaller than the .h5ad (198 against 228 MB). The “Which format?” section of the documentation’s preparation guide has the full table.
data (Any) –
An anndata.AnnData or a mudata.MuData (pip install "cellucid[mudata]" to read and build one) that fills every component below an explicit argument does not:
- AnnData: obs; X as gene_expression with var (neither when X is absent); every numeric obsm array as an embedding under its own key, except a <field>_<basis> array as wide as X_<basis>, which is that vector field; obsp['connectivities']; and from uns['cellucid'] the identity (id, name, description, citation, source) and the launch defaults. Layers, raw, other obsp matrices, obsm entries that are not numeric arrays, and obs/var columns of no AnnData encoding Cellucid writes (a date, a time span) are not written, and a message names each one, as cellucid-r names them (the same column given as obs= is refused, saying how to convert it). To leave out a component data holds, remove it from a copy first.
- MuData (modalities over shared cells, axis 0): the cells’ mdata.obs, mdata.obsm (the shared embeddings and vector fields), mdata.obsp['connectivities'] and mdata.uns['cellucid'] at the root, as above; each modality whole – its X, var, obs, numeric obsm arrays and obsp['connectivities'] – as mod/<name>. Columns and arrays mudata copies to the root from a modality (<name>:<column>) are not written twice: the viewer shows every modality’s own under that name. The genes are each modality’s own, so gene_expression, var and gene_identifiers are refused with a MuData: change mdata.mod[name] first.
embeddings (Mapping[str, Any] | None) – {obsm key: (n_cells, k) array}, at least one (with a MuData, at the root or in a modality). Each is stored whole as obsm[key] in float32, every column, at its own coordinates (the viewer normalizes them). A latent space for outlier measurement is one of them, named by latent_key. A plain (n_cells,) vector is one column; sparse input is densified. A row holding NaN (or an infinity) is a cell the embedding does not place (one computed on some of the cells): it is written as given, the viewer leaves it out of that embedding’s views, and in the latent space it gets no outlier quantile; at least one cell must have finite values. X_umap_1d/X_umap_2d/X_umap_3d, else the first 1- to 3-column of X_umap, X_UMAP, X_wnn.umap and X_ref.umap, are shown by default.
dataset_id (str | None) – The dataset’s stable id: 1 to 180 ASCII letters, digits, ., _ or -, starting with a letter or digit and not ending with .. Sessions and community annotations are keyed by it.
dataset_name (str | None) – The name shown for the dataset, also stored as uns['title'].
obs (DataFrame | None) – Cell metadata, one row per cell. Its index names the cells (pandas’ default 0 .. n-1 index names them "0" to "n - 1"); repeated names are made unique, as anndata does. Text columns become categoricals when a value repeats or is missing (anndata’s rule); numbers, booleans, categoricals and pandas nullable integers and booleans are stored as they are. Infinities are refused.
var (DataFrame | None) – Gene metadata, one row per column of gene_expression. Its index names the genes; with pandas’ default index, var_gene_id_column names them. Every column is stored.
gene_expression (ndarray | spmatrix | sparray | None) – (n_cells, n_genes), sparse (any scipy format) or dense. Stored as X: CSC when sparse, dense otherwise, in float32 – or float64 when the genes it keeps (after gene_identifiers) hold integer counts above 2^24, which float32 cannot hold exactly. Values must be finite. A stored zero is dropped.
connectivities (ndarray | spmatrix | sparray | None) – (n_cells, n_cells) neighbour graph, stored as obsp['connectivities'] (CSR float32). Exactly symmetric, finite, non-negative; its diagonal is left out (a self-loop is no edge: Seurat’s RNA_snn has 1 there), and a directed graph (RNA_nn, distances) is refused. From an AnnData data=, the graph the viewer’s default rule opens with: obsp['connectivities'], else Seurat’s <assay>_snn.
vector_fields (Mapping[str, Mapping[str, Any]] | None) – {field: {embedding key: (n_cells, k) array}}, one column per column of the embedding. Each array is stored as obsm['<field>_<basis>'] for the embedding X_<basis>, scVelo’s name ({"velocity": {"X_umap": v}} writes velocity_umap), which the viewer draws as the field <field> over that embedding, so the embedding’s key must start with X_.
modalities (Sequence[str] | None) – With a MuData, the modalities to write, in this order (names of mdata.mod); None writes every one, in mdata.mod order. Several are written as a Cellucid-ready MuData of those modalities (the root var and varmap hold theirs alone). One is written as a plain Cellucid-ready AnnData (format="h5ad", "zarr" or "zarr.zip"): the cells’ obs, embeddings, graph and uns['cellucid'] from the root, and that modality’s X, var and obsm arrays, its rows in the MuData’s cell order – so it must have measured every cell – with gene_expression, var and gene_identifiers then applying to it. cellucid-r’s cellucid_prepare(modalities = ) takes the same names.
gene_identifiers (Sequence[str] | None) – Keep only these genes, in this order, matched against the names the viewer shows (var_gene_id_column, else the var index).
dataset_description (str | None) – One paragraph about the dataset.
citation_text (str | None) – How to cite the dataset; the address needs the text.
citation_url (str | None) – How to cite the dataset; the address needs the text.
source_name (str | None) – Where the data comes from; the address needs the name.
source_url (str | None) – Where the data comes from; the address needs the name.
embedding_defaults (Mapping[str, str | tuple[str, Sequence[int]]] | None) – The embedding each dimension opens on: {"2d": "X_tsne"} (its first columns) or {"2d": ("X_pca", [2, 0])} (0-based columns).
latent_key (str | None) – The embedding (an embeddings key) the viewer’s outlier filter measures in: a latent space such as scVI or PCA, never a 2D/3D layout. The outlier quantiles of every categorical obs column (each cell’s distance to its category’s centre, as the share of its category at that distance or closer) are computed here and stored in uns['cellucid']['outlierQuantiles'], so the viewer filters outliers at once instead of reading the whole latent space to compute them.
obs_keys (Sequence[str] | None) – The obs columns the viewer lists, in this order; every column is stored either way.
var_gene_id_column (str | None) – The var column whose values name the genes in the viewer. Omitted, the var index names them; Data mapping can choose a column later.
vector_field_default (str | None) – The vector_fields field the overlay opens on; one of its embeddings must be shown when the file opens (a chosen or default embedding, or a 1-3 column array that is not the latent space).
normalize_embeddings (bool | None) – False shows embeddings at their own coordinates.
centroid_min_points (int | None) – The fewest cells a category needs to get outlier quantiles (10 when None).
modality (str | None) – With a MuData, the modality the viewer opens on (one of mdata.mod); var_gene_id_column names a column of its var. When None the viewer opens on the first gene-expression modality (rna, gex, gene expression, gene_expression, sct, any case), else the first. Refused for an AnnData, which has one feature space.
priority_seed (int | None) – The seed of the cells’ priority order, an integer in [0, 2**32 - 1]. The cells are stored by ascending priority – the uniform rank of the subsampling specification (subsample_cells()) with this seed – so every subsample Cellucid opens with this seed is the first rows of every array, and reads nothing of the cells it leaves out. Each row’s original position is kept in obs['cellucid_original_position']; restore_original_order() puts the cells back in the order given. None stores the cells in the order given, without a priority.
zarr_format (Optional[Literal[2, 3]]) – 2 or 3, for format="zarr" or "zarr.zip"; Zarr v2 when None. Refused with format="h5ad" and "h5mu".
force (bool) – Replace an existing output of the same kind. A folder that is not a Zarr store is never replaced.
modality) (The launch defaults (embedding_defaults to)
choice (are written to uns['cellucid']['defaults'] only when given; a)
own. (left at None stays the viewer's)

Returns:

The written file or folder.

Return type:

Path

Raises:

TypeError – A required input is missing (dataset_id or dataset_name that data does not name, embeddings without data) or an input has the wrong type.
ValueError – An input is invalid or the inputs do not fit together (out_path not ending in the suffix of format, a MuData of several modalities with format="h5ad", an AnnData with format="h5mu", zarr_format with "h5ad" or "h5mu", …).
FileExistsError – out_path exists and force is False.

cellucid.restore_original_order(data)[source]#

The cells of a file written in priority order, back in the order they were given.

A Cellucid-ready file stores its cells by priority (prepare(), priority_seed), recorded in uns['cellucid']['priority'], each row’s original position in obs['cellucid_original_position'] (and, in a MuData, each modality’s rows’ in its own obs). anndata.read_h5ad and anndata.read_zarr (mudata.read_h5mu, mudata.read_zarr) give the cells in that stored order; this returns a new object with every per-cell element – obs, obsm, obsp (both axes), X, layers, and in a MuData every modality’s cells – in the original order, without the position column, uns['cellucid']['priority'] and varm['cellucid_prefix_counts'], which describe the stored order only; the stored outlier quantiles (uns['cellucid']['outlierQuantiles'], one value per cell) follow the cells. A backed AnnData (anndata.read_h5ad(path, backed="r")) is read into memory in that order.

data is returned as it is when its rows are each its own original position: it has no priority record, or one that no longer describes its cells – it was subset, concatenated or reordered after it was read (anndata keeps uns through all three) – which a warning reports. That is the rule the web app reads such a file by, so a session saved on it applies to data as it is.

Parameters:: data (Any) – An anndata.AnnData or a mudata.MuData.
Returns:: A new object in the original order, or data itself.
Return type:: AnnData or MuData

Examples

>>> adata = anndata.read_zarr("pbmc.zarr")
>>> adata = cellucid.restore_original_order(adata)

Edge cases#

Validation first. Every input is checked before anything is written; a refusal leaves nothing on disk.
Atomic placement. The output is written beside out_path under a temporary hidden name and moved into place only once complete.
Existing outputs. Replaced only with force=True; a folder that is not a Zarr store is never replaced.
Names. Repeated cell or gene names are made unique (MATR3, MATR3-1) and reported; keys and columns must be portable path components (Input requirements (global)).
Values. Expression and the graph must be finite and fit in float32; embeddings and vectors fit float32, a row of NaN being a cell the array does not place (written as given, not drawn in that embedding); obs/var numbers may hold NaN but not infinities.
Cell order. The cells are stored in priority order (priority_seed), so anndata.read_h5ad / anndata.read_zarr give them in that order; restore_original_order() puts them back in the order given. uns["cellucid"]["priority"] records the seed and the cell count (nObs) it was written for, and is verified before it is used: an object subset, combined or reordered since is taken as cells Cellucid did not write, with a warning (the specification’s “Verified before use”). An object in a verified stored order is put back in its original order before it is prepared again, whatever priority_seed is. Every result Cellucid hands back (sessions, selections, exports) names cells by their original positions and obs names.
Vector fields bind to an embedding named X_<basis>, with exactly as many columns as it, under a key that reads back as that field and embedding (Vector fields (velocity / displacement)).

Troubleshooting#

Every refusal, its cause and its fix: Troubleshooting prepare().

Data preparation (prepare)#