Forty analysis requests on three datasets: human DLPFC (Visium), mouse brain (MERFISH) and human breast cancer (Xenium). Thirty cover eleven analysis categories, from cell-type annotation and spatial domains to cell-cell communication and deconvolution; four are multi-step requests and six are boundary cases (one infeasible, three out of scope, two with a missing prerequisite).
STAT answered every request four times: Run 1 with Claude Sonnet 4.6, Runs 2–4 with DeepSeek V4.1 Flash at maximum reasoning effort (seeds 10, 20, 30). When STAT asked a question, a scripted user answered from the request's answer bank. Each answer was rated 1–5 by a Claude Opus 5.5 rater following a fixed rubric and per-request criteria; a rating of 3 or more counts as a success. The benchmark compared ten systems; this page lists STAT's runs.
| Condition | Run | Rating | |
|---|---|---|---|
| Q01Annotate cell types on this tissue. It is a mouse brain cortex sample. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q02Identify cell types in this breast cancer tissue sample. I have a single-cell reference dataset at './breast_cancer_reference.h5ad' with cell type annotations that you can optionally use. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q03What cell types are present in this tissue? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q04Can you identify the different cell populations here? I have a reference scRNA-seq dataset at './breast_cancer_reference.h5ad' if that helps. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q05Identify spatial domains in this brain tissue. There should be about 7 distinct regions including cortical layers and white matter. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q06Can you find the spatial structure in this tissue? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 4/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 4/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 4/5 · successful | Replay → |
| Q07Identify distinct tissue regions based on spatial gene expression. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q08Find genes that show significant spatial expression patterns across this tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q09Which genes have interesting spatial patterns? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q10Identify spatially variable genes in this mouse brain section. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q11Analyze ligand-receptor interactions between cell types in this mouse brain tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q12Find cell-cell communication patterns between tumor and immune cells in this human breast cancer sample. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q13How are the cells communicating with each other? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q14Test whether certain cell types tend to co-localize or avoid each other spatially in this mouse brain tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q15Analyze spatial co-localization patterns between cell types in this breast cancer sample. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q16Are certain cell types found near each other in this tissue? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q17Estimate cell type proportions for each spot in this Visium tissue. I have a single-cell reference dataset at './dlpfc_reference.h5ad' with cell type annotations. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q18What cell types make up each spot in this tissue? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q19Identify spatial microenvironments (niches) based on the local cell type composition in this mouse brain tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 4/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q20Find the spatial neighborhoods in this tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q21Find differentially expressed genes between the different spatial domains in this tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 4/5 · successful | Replay → |
| Q22What genes distinguish the tumor regions from the surrounding stroma? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q23Run gene ontology enrichment analysis on these marker genes: RELN, CUX2, PCP4, MBP, MOBP, NTNG2, TLE4. The species is human. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q24What biological pathways are active in the tumor microenvironment? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 4/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 4/5 · successful | Replay → |
| Q25Integrate these four brain tissue slices, correcting for batch effects while preserving biological variation. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q26These two tissue sections are from the same brain region but look quite different. Can you make them comparable? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q27Give me a summary of this dataset — how many spots, how many genes, what spatial extent, and what annotations are already present. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q28Show the top 10 most highly expressed genes across all cells in this tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q29Give me a complete analysis of this spatial transcriptomics data. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q30What can you tell me about this tissue? | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 4/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q31First annotate cell types, then analyze which cell types tend to co-localize in this mouse brain tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q32Identify spatial domains in this brain tissue (about 7 regions), then find genes that are differentially expressed between them. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q33Annotate cell types in this breast cancer sample using the reference at './breast_cancer_reference.h5ad', then analyze cell-cell communication between tumor and immune cells. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q34Identify cell types, find marker genes for each type, then run pathway enrichment on the tumor cell markers. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q35Estimate cell type proportions for each spot without any reference dataset. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 2/5 · not successful | Replay → |
| Q36Segment individual cells from the H&E image. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 4/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 4/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 4/5 · successful | Replay → |
| Q37Perform RNA velocity analysis on this spatial data. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q38Train a deep learning classifier to predict patient survival from this tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 4/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q39Identify spatial niches in this tissue based on cell type composition. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |
| Q40Test whether cell types co-localize spatially in this tissue. | |||
| STAT | Run 1 · Claude Sonnet 4.6 | rated 5/5 · successful | Replay → |
| STAT | Run 2 · DeepSeek V4.1 Flash · seed 10 | rated 5/5 · successful | Replay → |
| STAT | Run 3 · DeepSeek V4.1 Flash · seed 20 | rated 5/5 · successful | Replay → |
| STAT | Run 4 · DeepSeek V4.1 Flash · seed 30 | rated 5/5 · successful | Replay → |