File format expectations#

Audience: everyone (beginners get “what should I have?”, experts get exact file/key expectations)
Time: 10–15 minutes
What you’ll learn: which files Cellucid opens and how it reads each, what it expects of an .h5ad or an AnnData Zarr store, what a file says about itself in uns['cellucid'], and the exact datasets.json contract of a dataset catalog Prerequisites: none (helpful: you know which loading option you plan to use)

What Cellucid reads#

Every path (the file buttons, a dropped file, a link, the sample list, a server, a notebook) points the viewer at one of these inputs. Each is read as AnnData: whatever the file, data mapping, analysis, sessions, figure export and annotation work the same. Read in place means only the parts a view needs are read, from disk or with range requests from a link. The tag says which ecosystem’s tools write the format.

Input

Tag

Versions

How it is read

AnnData .h5ad

Python

anndata 0.6 and later

in place

AnnData Zarr: a .zarr folder, or .zarr.zip / .zip holding one

Python, R

Zarr v2 and v3, anndata 0.7 and later

in place

Loom .loom

any

Loom 2.0.1 and 3.0.0 (loompy, velocyto, SCENIC, LoomExperiment, loomR)

in place (Open Loom and 10x Cell Ranger files)

10x Cell Ranger .h5

any

Cell Ranger 1.2 and later

in place; alone it has no embedding, so open its outs/ folder (Open Loom and 10x Cell Ranger files)

Cell Ranger outs/ folder

any

Cell Ranger 1.2 and later

its .h5 in place, with analysis/ projections and clusterings (Open Loom and 10x Cell Ranger files)

MuData .h5mu

Python

mudata 0.2 and later

in place, with its modalities (Multimodal data: MuData and the Modality select)

MuData Zarr (a folder or .zarr.zip)

Python, R

mudata 0.2 and later, Zarr v2 and v3

in place, with its modalities (Multimodal data: MuData and the Modality select)

Seurat or SingleCellExperiment object: .rds, .rda, .RData, and their gzipped copies .rds.gz, .rda.gz, .RData.gz (GEO)

R

SeuratObject 3.0 and later (Assay, Assay5), SingleCellExperiment 1.8 and later, SummarizedExperiment; R serialization 2 and 3, gzip (once or twice), bzip2, xz or none

cannot be read in parts: read whole once into this browser’s storage and kept for next time; an uncompressed file’s arrays in place (Open R objects and MatrixMarket folders, Which formats are read in parts)

A gzipped copy of a file read in parts: .h5ad.gz, .loom.gz, .h5.gz, .h5mu.gz (GEO ships many .h5ad files so)

any

as the file it copies

cannot be read in parts (gzip has no index): read whole once, decompressed as it arrives into this browser’s storage and kept for next time, then read in parts from there; the uncompressed file is opened instead when the host has both. cellucid serve and cellucid_serve() refuse it with how to decompress it

MatrixMarket folder: a .mtx[.gz] file (matrix.mtx.gz, a GEO sample’s GSM…_matrix.mtx.gz, any producer’s name) with its features and barcodes (its own prefix’s lists)

any

Cell Ranger 2 and later, STARsolo, kb-python, alevin-fry, GEO uploads, sci-RNA-seq3 (_gene_count.mtx.gz, or GSE119945’s _gene_count.txt.gz, with _cell_annotate.csv.gz and _gene_annotate.csv.gz)

cannot be read in parts: its cells are drawn from its lists and tables first, then the matrix is read once keeping the rows of the cells genes are read for, both kept in this browser’s storage for next time, with the folder’s cell tables for the embedding (Open R objects and MatrixMarket folders)

A dataset catalog: a folder holding datasets.json and the files it lists

any

schema 2

below

The format is decided by the file’s content, not only its name: an .h5 that is a Loom or an AnnData file opens as what it is, and a file Cellucid recognises but cannot show (a Cell Ranger molecule_info.h5, a MuData whose modalities share features rather than cells, a count matrix with no embedding, an h5Seurat file, a Seurat object whose counts live outside the .rds) is refused with what to do instead (Troubleshooting (data loading)).

A Cellucid-ready AnnData is not a format of its own: it is an ordinary .h5ad or Zarr store that cellucid.prepare() (Python) or cellucid_prepare() (R) wrote for fast reading (the expression matrix gene-major, arrays chunked for range requests) with uns['cellucid'] filled in. The exact contract is Cellucid-ready AnnData specification; how to write one is in Preparing an AnnData for Cellucid (Python) and Data Preparation API (R). anndata reads it like any other file, and Cellucid reads any other file like it.

What a file says about itself: uns['cellucid']#

An AnnData file may carry a mapping uns['cellucid'] that tells Cellucid its identity and the choices it opens with. Every key is optional, and a file without the mapping opens exactly as any AnnData file does.

Key

Value

Used for

schema

the integer 1

the version of this mapping; a later version is refused, asking for a newer Cellucid

id

the dataset’s stable id: 1–180 ASCII letters, digits, ., _ or -, starting with a letter or digit, not ending with .

the dataset id sessions and community annotations key on (Dataset identity (why it matters))

name

text

the name shown in the dataset list and the Dataset box

description

text (may be empty)

the Dataset box’s description

citation

a mapping {text, url} (url optional)

how to cite the dataset (the Citation row of Details)

source

a mapping {name, url} (url optional)

where the data comes from (the Source row of the Dataset box’s Details, linked to url)

defaults

a mapping of launch defaults (below)

the choices the dataset opens with

outlierQuantiles

a mapping {latentKey, minPoints, fields}: the latent space and category size they were computed with, and one float32 array per categorical obs column (NaN where a cell has none), written by prepare(latent_key=...) / cellucid_prepare(latent_key = ...)

the outlier filter, which reads one column’s array on its slider’s first move instead of computing it from the latent space (Outlier filtering (per categorical field))

priority

a mapping {seed, specVersion, nObs, levels}: the file stores its cells in priority order, as prepare() and cellucid_prepare() write them

reading the first cells of a large file first; a record that does not describe the file’s cells is not used, never refused (Cellucid-ready AnnData specification)

history_applied

text: the records apply_cellucid_session_to_anndata() (R: cellucid_apply_session(); the web app’s Save AnnData with this state…) appends, one per apply that wrote something (a JSON list)

never read by the web app

gene_sets

a mapping by gene set name: the gene sets a pull, an applied history or Save AnnData with this state… wrote

never read by the web app

pulled

a mapping by column name: what each column a pull wrote came from

never read by the web app

Rules the reader applies:

  • Text is stored as text; a one-element list of text is read as that text. It must not start or end with white space or hold a character with no glyph (a control or zero-width character).

  • An address (citation.url, source.url) is an absolute http(s) URL.

  • A key outside this table, or a wrong value, refuses the file with a message naming it, so a misspelt default never passes unnoticed.

  • history_applied, gene_sets and pulled are the records the Python and R packages keep on the object a person works with, and that the web app writes into a copy saved with the History header’s ⋯ › Save AnnData with this state… (Save AnnData with this state), so that object or file opens again in the viewer after a pull or an apply.

defaults holds the launch defaults, each optional: embedding1d, embedding2d, embedding3d (an obsm key, or {key, columns} with 0-based columns), knn (an obsp key), modality (for a file of several modalities, the one genes are read from), expression (X, raw/X or layers/<name>, of that modality), geneIdColumn (a var column), latentKey (an obsm key), vectorFieldDefault (a vector field id), obsKeys (a list of obs columns), normalizeEmbeddings (true or false), centroidMinPoints (a positive integer: the fewest cells a category needs to get outlier quantiles). These are the launch choices cellucid serve, serve_anndata() and show_anndata() take (Server Mode (CLI + Python)).

How the file’s own values combine with the route that opens it:

  • Id: the file’s id is the dataset id. A catalog lists the file under an id, which must equal the file’s (else the file is refused); a server that names an id serves the file under it.

  • Name: for a file you open yourself (from disk, a link or a catalog), the file’s name is shown, else its uns['title'] (as CELLxGENE writes it), else the file name. A server that names the dataset shows its name.

  • Description, citation, source: the file’s, whatever route opens it.

  • Launch defaults: the file’s, with every choice a server names on its command line taking precedence.

Beside uns['cellucid'] and uns['title'], the one other part of uns Cellucid reads is the colours Scanpy keeps for a categorical obs column, uns['<column>_colors']: the column opens in them (Color-by behavior). They are read with the column’s categories when the column is first shown: opening a file reads no category list, only how many categories each column has, so a column of one category per cell (barcodes stored as a categorical) costs nothing until it is shown. A column with more categories than there are open cells, and more than one request of the connection reads, is read codes first, then the names and colours of the categories those codes name alone: a share of the cells reads a share of its categories (1 % of a 1.2M-category column, about 12,000 names, not 35 MB).

A dataset catalog: datasets.json#

A dataset catalog is a folder holding datasets.json and the files it lists. The Sample dataset list reads one (www.cellucid.com reads https://theislab.github.io/cellucid-datasets/anndata/), and so does a link to a folder (Open from a link).

{
  "schema": 2,
  "default": "pancreas",
  "datasets": [
    {
      "id": "pancreas",
      "path": "pancreas.zarr/",
      "session": {
        "path": "pancreas.cellucid-session",
        "bytes": 3422,
        "sha256": "14eeb50c90eacd56a78704b2dd03756fb8147cdc2dfc7754d4059d6f6d7753c1"
      }
    },
    { "id": "suo", "path": "suo.h5ad" }
  ]
}

  • schema is the integer 2. A catalog without it is refused, naming the three keys a catalog must hold.

  • default is the id of the dataset the catalog opens on.

  • datasets lists one dataset or more. Ids are distinct, and so are paths. No other key is read; an unknown key is refused.

  • id: the dataset’s stable id, with the same rule as uns['cellucid']['id']. When the file states an id, it must be this one.

  • path: where the dataset is, relative to the folder: a store folder (ending in /: an AnnData or MuData Zarr store), or a file whose name a format in the table above names (an .h5ad, a .zarr.zip, an .h5mu, a Seurat .rds, …). Each part is written in canonical URL encoding (as encodeURIComponent writes it, e.g. cell%20atlas.h5ad), without ., .., a query or a fragment, and the path stays below the folder.

  • session (optional): the dataset’s default view, a published .cellucid-session file — a session file of the view passed through the publish step, which keeps one tracker of one root (no step, no cells to open, no layout) and binds it to the catalog’s dataset id (see Starting views named by a catalog) — with its exact byte length (bytes) and lowercase SHA-256 (sha256). It is applied when the dataset is opened from the catalog (Starting views named by a catalog). The length and digest pin it to the catalog: a session file that differs is refused rather than applied.

The dataset list shows each dataset’s name and cell count, read from its file’s metadata and uns alone (no matrix, column or index is read); the dataset’s own file is opened when it is chosen. A Zarr store folder on a web host needs consolidated metadata (.zmetadata for Zarr v2, the root zarr.json of v3), which prepare() writes.

To compute bytes and sha256 of a session file:

import hashlib
from pathlib import Path

data = Path("pancreas.cellucid-session").read_bytes()
print(len(data), hashlib.sha256(data).hexdigest())

The worked public example is Publish a Custom Dataset Repository; the shorter publishing workflow is Share Without Running a Server (Publish a Dataset Catalog).

AnnData H5AD and Zarr (what Cellucid reads)#

The browser reads one H5AD file, one ZIP containing a complete Zarr store (Zarr v2 or v3, sharded or not), a Zarr store folder from your disk (the Open… › Folder…, or a dropped folder), or a Zarr store folder on a web host (through its consolidated metadata). Server mode and Jupyter accept H5AD paths, Zarr store folders (v2 or v3), zipped Zarr stores, an in-memory AnnData object, and every other input of the table above. The server serves a file prepare() wrote as it is on disk, and the browser reads it with the same reader; an in-memory AnnData, and an AnnData, MuData, Loom or 10x Cell Ranger file (or outs/ folder) not in priority order, are served as a Zarr store in priority order made as the browser reads it, holding what the browser presents from the file itself (a MuData object is written once to an .h5mu). R objects and MatrixMarket folders given by path are served as they are and read once by the browser.

Compression the browser reads#

  • H5AD: uncompressed, gzip (with or without shuffle), LZF (h5py’s own filter), Fletcher-32 checksums, and hdf5plugin’s Blosc, Zstandard, LZ4 and bitshuffle (uncompressed, LZ4 or zstd blocks) filters (write_h5ad(compression=hdf5plugin.Blosc(...)) and the like). A dataset compressed with a filter outside these (bzip2, SZIP, N-bit, scale-offset, Blosc2) is refused when it is read, naming the filter; the rest of the file still opens.

  • Zarr: uncompressed, gzip, zlib, Blosc (blosclz, LZ4, LZ4HC, zlib, Zstandard; byte and bit shuffle, and the delta flag of a Blosc frame), LZ4, Zstandard (Zarr v3’s default), bz2, LZMA (numcodecs’ default xz container), CRC-32C, variable-length strings, and Zarr v3 sharding. An array using another codec (Snappy, ZFP, numcodecs’ Delta filter, fixed-scale-offset, …), or Blosc’s Snappy or a Blosc2 chunk (format version 3 and later), is refused when it is read, naming the codec.

  • A Zarr folder on Amazon S3 or Google Cloud Storage: a bucket that does not let its readers list it answers “403 Forbidden” for an object that does not exist, and zarr-python leaves out every chunk that holds only the fill value (an all-False mask, an all-zero block). Once the bucket has served the store’s metadata, such a 403 is read as the absent chunk it is (its fill value); a bucket that refuses the metadata itself is private, and the store is refused saying so.

  • Served from Python: cellucid serve, serve_anndata() and show_anndata() check every array’s filters and codecs before they start. A store holding an array the browser cannot decode is decoded on the kernel, chunk by chunk as the browser asks (h5py or zarr, with their codec plugins); the terminal names the arrays.

Minimum requirements (every loading route)#

You must have at least one obsm array Cellucid can draw: dense, numeric, one row per cell, at least one column (a one-dimensional array is one column), with finite values that fit in 32-bit floats and not the same coordinates for every cell.

Which array each dimension shows by default:

  • obsm["X_umap_1d"], obsm["X_umap_2d"] and obsm["X_umap_3d"] fill their own dimensions; a suffixed key and its column count must agree exactly.

  • When a file declares none of the three and carries the plain X_umap that sc.tl.umap() writes, that array fills the dimension of its width (1, 2 or 3 columns), so an ordinary Scanpy object loads as written.

  • When the file has no UMAP at all, a deterministic best guess fills the dimensions instead (t-SNE, force layouts, spatial, …, and last the first two columns of a wide array such as X_pca). See Data mapping.

This holds for every loading route: the browser file picker, cellucid serve, and the notebook viewers. In the browser any other array can then be turned on under Session › Data mapping and shown in any view; cellucid serve takes --embedding-1d/--embedding-2d/--embedding-3d KEY[:c0,c1].

Files from older AnnData releases#

Files written before anndata 0.8 (2022) open as written, from the file pickers, a dropped file, a link and cellucid serve alike:

  • anndata 0.7, and every file SeuratDisk’s Convert() writes: dataframes whose categorical columns keep their categories under __categories, arrays without encoding attributes, and (Zarr) a store without a root .zattrs;

  • anndata 0.6 and earlier: obs and var as one record per cell (compound datasets) with categories in uns, a compound obsm, raw stored as raw.X / raw.var, and sparse matrices named by h5sparse_format.

Every column a file stores is shown, including those a SeuratDisk file leaves out of its column-order (anndata itself drops those). Ordered categories and missing values keep their meaning. Nothing needs converting first.

A column the browser cannot read#

One obs or var column the browser does not understand never refuses the file: a column of a compound, complex or opaque type, an encoding a later anndata introduced, the wrong length, or a column-order entry the frame does not hold is left out, and the rest of the dataset opens. The Dataset box’s Details list it under Not read, with the reason (“obs column ‘qc’ uses an unsupported element type; the column was not read.”). The same holds for raw.var and each MuData modality’s var. Opening looks at each column itself, not at the parts a categorical or nullable column is stored in (its codes and categories, its values and mask: read when the column is), so a categorical with malformed codes is listed and refused when it is chosen, with the same reason, while the rest of the dataset reads. Categorical codes may be unsigned integers, as some writers store them (anndata writes signed ones, with -1 for a missing value). What a file needs is its index, its shape and X; a defect there still refuses it.

Integers stay exact#

An integer column wider than 16 bits (int32, int64, uint32, uint64, a nullable Int64) is held as float64: every integer up to 2^53 stays exact, so cell or donor ids and deep cells’ counts above 16,777,216 are never merged by rounding, in colours, filters, highlights, analyses and exports. Narrower integers and every float column are held as float32, as before. prepare() and show_anndata(adata) take an AnnData you have already read, so they hold the columns anndata read; pass the file’s path to cellucid serve, serve_anndata or show_anndata to keep the unlisted ones.

Optional (but commonly expected)#

  • Obs fields (metadata for coloring/filtering): adata.obs[...]

  • Gene metadata (for gene search / naming): adata.var[...] and/or adata.var.index

  • Gene expression: adata.X (dense or sparse)

  • Connectivity: adata.obsp["connectivities"] (KNN graph)

Vector fields (optional overlays)#

Vector fields are optional per-cell displacement vectors (e.g. RNA velocity, drift) that Cellucid can render as an animated overlay.

If you expect vector fields, the key requirements are:

  • Correct width: exactly as many columns as the embedding the vectors are drawn over.

  • Correct row order: vectors must match the same cell order as points/embeddings.

  • Discoverable naming / metadata: so the viewer can find the fields.

See the overlay UI docs:

AnnData naming convention (obsm)#

Cellucid discovers vector fields from adata.obsm keys using scVelo’s convention:

  • <field>_<basis>: the field <field> drawn over the embedding X_<basis>

Examples:

  • velocity_umap over X_umap, velocity_umap_3d over X_umap_3d

  • T_fwd_umap, T_bwd_umap over X_umap

  • velocity_pca over X_pca, velocity_branches over X_branches

Notes:

  • A key that starts with X_ (X_umap, X_umap_3d) is an embedding, never a vector field.

  • The key is split at one _ into a field and a basis whose X_<basis> the file holds; when several splits match, the longest basis wins. A key with no match is an ordinary obsm array, so velocity_umap_2d beside a bare X_umap (no X_umap_2d) is not a vector field: name it velocity_umap.

  • A field is bound to every enabled view of X_<basis> when both arrays have the same number of columns. The keys of one field on every embedding are one field: velocity_umap and velocity_umap_3d are both velocity.

  • The overlay dropdown shows the fields bound to the embedding the view shows; a view showing another embedding (for example a t-SNE) has none.

How to confirm vector fields exist (quick checks)#

  • Print adata.obsm.keys() and look for keys like velocity_umap beside X_umap.

Naming and schema details:

Common “format mismatch” problems (and what they mean)#

“Cellucid found nothing to plot”#

Meaning:

Fix:

  • Add an embedding (for example sc.tl.umap(adata)) or fix the array the message names. A file without a UMAP but with any drawable array opens on a best guess (Data mapping).

“Fields list is empty”#

Meaning:

  • obs is empty, or uns['cellucid']['defaults']['obsKeys'] names only some columns.

Fix:

  • Ensure adata.obs actually contains columns and is not all missing.

“Gene search returns nothing”#

Meaning:

  • There is no gene expression (X), or the genes were named from a var column whose vocabulary nobody is searching in.

Fix:

  • Provide adata.X and name the genes from the var column your readers will type.

  • In Jupyter/server mode, pass gene_id_column=... if needed.

Edge cases and limits (read if you’re debugging weirdness)#

  • NaN/Inf in embeddings: a cell whose row holds NaN or an infinity has no position in that embedding (one computed on some of the cells, a MuData modality that did not measure it). The embedding opens; the cell is not drawn, picked or selected in its views, a short notice counts such cells and names the first, and every other view, filter and analysis keeps them. In a latent space such a cell gets no outlier quantile; in a vector field it has no arrow. Only an embedding with no finite row, or a value past float32, is refused, naming the cell.

  • Duplicate gene names: made unique as AnnData’s var_names_make_unique() does (MATR3, MATR3-1), and reported.

  • Huge categorical fields: any number of categories loads and the legend scrolls, but palettes cannot keep 50k–100k colours distinct; consider aggregating.

  • Browser memory: there is no fixed size ceiling for an .h5ad or Zarr file. Each array read is held to the size its metadata declares, and one the browser cannot allocate is refused as too large, naming the bytes. A server hands the file to the browser, which reads only what it asks for.

  • Byte order: never a compatibility problem. H5AD is byte-order-agnostic, because HDF5 converts to host order while reading; Zarr records its own byte order per array and the reader swaps when it has to.

Next steps#