# chromoMap2 batch reducer

An optional Python 3.9+ companion for batch processing on a workstation or HPC.
The browser app remains usable without installing this tool. No browser-versus-native
speedup or hundreds-of-samples benchmark is claimed. Parallel sample processing is
bounded by `--workers`; performance depends on CPU, RAM, storage and sample size.

## Requirements

- Python 3.9 or newer (standard library only).
- BAM: native **Samtools** on PATH, coordinate-sorted BAMs and matching
  `sample.bam.bai` or `sample.bai` files in the same directory.
  Use the browser's Samtools version (1.21) when strict cross-version parity matters.
- BigWig only: UCSC **bigWigToWig** on PATH. The browser uses Kent tools version 437.
- Plain or gzip/BGZF GFF3 reference. An annotation index is not needed for a full scan.
- BED, bedGraph, numeric CSV/TSV/TXT and BAM inputs use the same reference/assembly.
  This tool cannot infer or prove assembly compatibility from identifiers alone.

## Installation on a local machine

The reducer is a standalone Python file, not a pip package. There is no `pip install`
step and no Python-library dependency. Use an isolated environment when BAM/BigWig
executables are needed. These commands are installation instructions, not a claim
that every OS/architecture combination has been tested.

### Linux or macOS: Conda environment

1. Install Conda/Miniforge for your OS and CPU, or use an existing Conda installation.
   Follow [Miniforge's official installer instructions](https://github.com/conda-forge/miniforge).
   Open a new terminal where `conda --version` succeeds.
2. Download `chromomap-reducer.zip` from the app and extract it. In a typical Downloads
   folder (change the path if you downloaded elsewhere):

```sh
cd ~/Downloads
python3 -m zipfile -e chromomap-reducer.zip .
cd chromomap-reducer
conda create --name chromomap-reducer --override-channels \
  -c conda-forge -c bioconda --strict-channel-priority python=3.11 samtools=1.21
conda activate chromomap-reducer
python --version
samtools --version
python chromomap_reduce.py --help
```

If `python3` is unavailable for extraction, use your file manager's Extract action;
Python becomes available after environment creation. The environment and the
extracted folder share a name but are separate: activating the environment does not
change your working directory. Run the script from its extracted folder, or supply
its full path. Use `python` after activation so the environment's interpreter is used.

For **BigWig only**, add the optional native converter:

```sh
conda install --name chromomap-reducer --override-channels \
  -c conda-forge -c bioconda --strict-channel-priority ucsc-bigwigtowig
command -v bigWigToWig
```

Package availability can depend on architecture. If the converter cannot be installed,
use a supported Linux environment or convert BigWig to a supported numeric format
with your existing validated workflow. Do not infer BigWig parity from the BAM tests.
See [Bioconda installation guidance](https://bioconda.github.io/).

### Existing Python/Samtools installation

For gene density and numeric text files, Python 3.9+ alone is sufficient:
`python3 /path/to/chromomap_reduce.py --help`. For BAMs, `samtools --version` must
succeed in that same terminal. This installation route was used for the local tests
(Python 3.9.7, native Samtools 1.23.1 on macOS); browser Samtools 1.21 was also tested
for index validation/depth. Conda setup itself was not executed as part of those tests.

### Windows

Use WSL2 with a Linux distribution such as Ubuntu and run the Linux commands inside
that shell; see [Microsoft's WSL installation instructions](https://learn.microsoft.com/windows/wsl/install).
Copy the ZIP and input data into a directory accessible from WSL and adjust the paths.
A native Windows installation and WSL execution have not been validated here.

### Quick installation check using the supplied data

Download `reduction-demo.zip` from the app, extract it into a separate `demo` folder,
and run this from the reducer folder in the activated environment:

```sh
python -m zipfile -e /path/to/reduction-demo.zip demo
python chromomap_reduce.py --reference demo/reduction-reference.gff3 \
  --mode gene-density --window-size 10000 --output demo-density
```

Expect 300 output windows and 6,000 counted genes, with per-window values matching
`demo/reduction-expected.json`. Inspect the results and manifest before using real data.
Choose a new output directory on each run; existing outputs are protected.

## HPC installation and execution

Use your cluster's documented Conda/module setup; module names and policies vary.
Create the environment once on a permitted setup/login node (not inside every batch
job), or ask the administrator for Python and Samtools modules. Inputs, the environment
and output paths must be accessible on the compute node. Network access is not needed
during reduction after dependencies and input files are in place.

The download includes `example.slurm`, a single-node Slurm template. Edit all
`/absolute/path/...` placeholders, resource requests and account/partition settings
for your cluster, then submit with `sbatch example.slurm`. Its 4 CPUs, 16 GB and
4-hour limits are examples, not measured requirements. `--workers` controls sample
processes on one node; this tool is not a distributed/multi-node program. For other
schedulers, launch the same Python command in the allocated job and set `--workers`
no higher than the CPUs allocated. Start with a small batch: disk bandwidth and RAM
can limit useful concurrency. Log paths and sample processing times help tune it.

## Examples

```sh
python chromomap_reduce.py --reference reduction-reference.gff3 \
  --mode gene-density --window-size 10000 --output density

python chromomap_reduce.py --reference reduction-reference.gff3 \
  --mode numeric-regions --inputs reduction-sample-1.bedgraph reduction-sample-2.bedgraph \
  --window-size 10000 --summary avg --workers 2 --output numeric

python chromomap_reduce.py --reference annotation.gff3.gz \
  --mode bam-depth --input-list samples.txt --window-size 100000 \
  --summary avg --workers 8 --output depth
```

`samples.txt` has one BAM path per line. Relative paths resolve against the list's
folder. Blank lines and lines beginning with `#` are ignored. Indexes sit beside BAMs.
Start with a few workers; hundreds of simultaneous jobs can saturate storage or RAM.
An existing output directory is rejected so previous results are preserved.

## Output and import

- `chromosomes.tsv`: upload as the chromosome table for one set in the app.
- `annotations.tsv`: upload as annotations for that set. Sample columns are `data`,
  `data2`, `data3`, etc.; blanks mean no populated window, not measured zero.
- `sample-N-windows.tsv`: each sample's windows, values and record counts.
- `manifest.json`: options, sample order/names, actual window size, processing times.

Choose Numeric in the app, fixed resolution using `window_size` from the manifest,
aggregation None, and multi-line or multi-area. Add sample legend labels in manifest
order. Use smaller batches or subsets when hundreds of curves would be unreadable.

## Summary semantics and parity

Gene density counts each gene's midpoint once. Numeric intervals contribute one
record to every intersected window, without weighting by interval length. Tables
use positive 1-based inclusive coordinates; BED/bedGraph are 0-based half-open.
Headered numeric files need chromosome (`ch_name/chrom/chromosome/seqid`), start
(`ch_start/start/position/pos`), value (`data/value/score`) and optional end
(`ch_end/end`). Browser and CLI use the same aliases and chr-prefix reference aliases.

BAM processing streams `samtools depth`, i.e. **depth at each nucleotide** using
Samtools' default read filters. Average is the mean over covered positions. Zero-depth
positions are omitted, matching the browser; this is not a whole-window mean including
uncovered bases. Count counts covered nucleotide positions, not reads or depth. Sum
adds per-base depths. Average is the default for numeric/depth; count is the default
for gene density. The tool does not output a raw per-base table unless window size is 1.

`--window-mode adaptive --window-factor 1` follows the browser's shared grid using
longest sequence length / (100 × factor), with a minimum 1-bp step. The grid origin is
the minimum reference start. All samples use that same grid. Supported statistics:
count, sum, avg, min, max, median. Median uses the browser's deterministic bounded
2048-record reservoir, so larger-window medians are approximate.

Automated parity tests compare GFF3, numeric tables and native BAM depth summaries
against the browser reducers. BigWig uses the same converter family and WIG parsing
semantics but is not covered by a BigWig parity fixture yet. Ordering and floating-point
formatting can differ; values are compared numerically. CPU-intensive Python reduction
is not automatically faster than C++/WASM; native Samtools and multiple sample processes
provide a batch execution path, not a demonstrated speed guarantee.

This is a companion download, not a separately validated NAR stand-alone application.
No cross-platform or container execution certification is claimed. Retain the project's
CC BY-NC 4.0 software license and attribution.