# LOCK PIPELINE

This workspace computes lock behavior (modulo 19) across 10 Quran datasets (`*_7Metrics.xlsx`) using three core scripts:

- `verse_locks.py`
- `sura_locks.py`
- `global_lock.py`

A fourth helper script is included to build the combined summary automatically:

- `locks_summary.py`

An orchestrator script runs all four stages in the correct order with one command:

- `run_pipeline.py` — **recommended entry point**

A hash registry script builds and verifies SHA-256 digests of the current workspace state:

- `hash_registry.py` — run after each pipeline execution to capture or verify file state across inputs, scripts, docs, config, and generated outputs

A shared internal utility module is required by all four scripts:

- `_pipeline_utils.py` — provides the single canonical `_generated_timestamp()` implementation; **do not delete or rename this file**

## Reproducible Environment

Use Python `3.11+` (tested in this workspace with `3.15.0a6`) and install pinned dependencies:

```powershell
python -m venv .venv
.venv\Scripts\activate
python -m pip install -r requirements.txt
```

Pinned dependency file:

- `requirements.txt`

## Input Data Contract

Each workbook must contain one valid worksheet with:

- `sura` column
- `verse` column
- `text` column
- 7 metric columns corresponding to `M1..M7` (for example `M1_LetterCount`, `M2_WordCount`, etc.)

Metric handling rules:

- `M1`, `M2`, `M3`, `M5`, and `M7` are read directly from workbook metric columns.
- `M4` is computed in code from the `text` column as forward letter-abjad concatenation.
- `M6` is computed in code as the reverse-letter mirror of `M4`.
- Text is normalized with Unicode NFKC before extraction.
- Arabic combining marks and Quranic diacritics are ignored.
- The following character remaps are applied before abjad lookup: `أ/إ/آ/ٱ -> ا`, `ى/ئ -> ي`, `ؤ -> و`, `ة -> ه`, `ء -> ا`.

## Script 1: `verse_locks.py`

Purpose: Evaluate verse-level metric values.

What it does:

1. Scans each verse row in each workbook.
2. Uses workbook values for `M1`, `M2`, `M3`, `M5`, and `M7`, and text-derived abjad values for `M4` and `M6`.
3. For each metric `M1..M7`, checks whether the verse value is divisible by 19.
4. Counts lock hits per metric and total lock hits.
5. Writes detailed verse-level hits (`mushaf | sura:verse -> Mx=value`).

Output files:

- `summary/txt/grand_verse_lock_summary.txt`
- `summary/txt/grand_verse_lock_summary_bool.txt`
- `summary/json/grand_verse_lock_summary.json`

Main measurements reported per dataset:

- `Verses scanned`
- `Total metric locks`
- `Locks by metric (M1..M7)`

## Script 2: `sura_locks.py`

Purpose: Evaluate sura-level aggregated metric values.

What it does:

1. Aggregates each metric by sura (sum of all verse values in that sura).
2. Uses text-derived `M4` and `M6` values during aggregation; all other metrics come from workbook values.
3. For each sura and each metric, checks whether the sura-total is divisible by 19.
4. Counts lock hits per metric and total lock hits.
5. Writes detailed sura-level hits (`mushaf | sura N -> Mx=total`).

Output files:

- `summary/txt/grand_sura_lock_summary.txt`
- `summary/txt/grand_sura_lock_summary_bool.txt`
- `summary/json/grand_sura_lock_summary.json`

Main measurements reported per dataset:

- `Suras scanned`
- `Total metric locks`
- `Locks by metric (M1..M7)`

## Script 3: `global_lock.py`

Purpose: Compute global lock families from full-dataset streams and sura aggregates.

What it does:

1. Enforces canonical metric order: `M1..M7`.
2. Recomputes `M4` and `M6` from verse text before all global measurements; `M1`, `M2`, `M3`, `M5`, and `M7` remain workbook-sourced.
3. Computes, per metric, six measurement families (`A..F`):

- `A`: Global verse sum (single value)
- `B`: Forward verse concatenation (all verse values in sura/verse order)
- `C`: Reverse verse concatenation
- `D`: Global sura sum (sum of 114 sura totals)
- `E`: Forward sura concatenation (114 sura totals)
- `F`: Reverse sura concatenation

4. Computes final family `G`:

- `G`: Concatenation of the seven `D` values in metric order `M1..M7`.

5. For every measurement, evaluates `mod 19` and marks `PASS` when remainder is 0.

Output files:

- `summary/txt/global_lock_summary.txt`
- `summary/txt/global_lock_summary_bool.txt`
- `summary/json/global_lock_summary.json`

Global measurement count:

- `43` total per dataset = `6 families * 7 metrics + 1 composite family G`

## Combined Summary: `locks_summary.py`

Purpose: Build a single consolidated report from the three generated summaries.

What it does:

1. Reads:
   - `summary/json/grand_verse_lock_summary.json`
   - `summary/json/grand_sura_lock_summary.json`
   - `summary/json/global_lock_summary.json`
2. Produces a per-dataset combined block with:
   - Verse lock totals
   - Sura lock totals
   - Global pass/fail totals and family pass counts (`A..G`)
3. Produces deterministic ranking using:

`0.60*GlobalPassRate + 0.25*SuraLockDensity + 0.15*VerseLockDensity`

where:

- `GlobalPassRate = global_passes / 43`
- `SuraLockDensity = total_sura_locks / suras_scanned`
- `VerseLockDensity = total_verse_locks / verses_scanned`

Output files:

- `summary/txt/LOCKS.txt`
- `summary/txt/LOCKS_bool.txt`
- `summary/json/LOCKS.json`

## Execution Order

### Recommended: single command via the orchestrator

```powershell
python run_pipeline.py
```

`run_pipeline.py` calls all four stages in the correct sequence, pins a **single shared timestamp** across all output files for the run, and creates the `summary/` directory structure if it does not already exist.

### Alternative: run scripts individually

The three core scripts (`verse_locks.py`, `sura_locks.py`, `global_lock.py`) are **independent** and can execute in any order relative to each other. `locks_summary.py` must run **last** — it reads the JSON outputs produced by the other three.

```powershell
python verse_locks.py
python sura_locks.py
python global_lock.py
python locks_summary.py
```

## Deterministic Output Mode

When run via `run_pipeline.py`, a single wall-clock timestamp is automatically captured once at startup and shared across all four output files, so all `Generated:` lines are identical for a given run.

For **bit-reproducible output across machines and reruns** (same bytes every time), pin the timestamp before running:

```powershell
$env:LOCK_PIPELINE_DETERMINISTIC = "1"
python run_pipeline.py
```

or set an exact fixed string:

```powershell
$env:LOCK_PIPELINE_GENERATED_AT = "1970-01-01 00:00:00"
python run_pipeline.py
```

When running scripts individually, set the same env var before all four invocations to keep timestamps consistent across output files.

## Notes on Strictness

- `global_lock.py` now enforces expected dataset scope and worksheet schema validation before computing results.
- `verse_locks.py` and `sura_locks.py` now enforce the same strict dataset scope and worksheet schema checks.
- Any missing/extra dataset or ambiguous worksheet schema is treated as an error rather than silently proceeding.

## Hash Registry

After each pipeline run, record the state of all files:

```powershell
python hash_registry.py
```

Verify that tracked workspace files have not changed since the last recorded state:

```powershell
python hash_registry.py --verify
```

Exit codes: `0` = all entries match; `1` = differences found; `2` = no registry exists yet.

Tracked by the registry:

- input workbooks (`*_7Metrics.xlsx`)
- pipeline scripts (`*.py`)
- documentation (`README.md`, `VALIDATION_REPORT.txt`)
- configuration (`requirements.txt`, `*.code-workspace`)
- generated summaries (`summary/**/*.txt`, `summary/**/*.json`)

When `--verify` is run after a pipeline rerun, expect tracked outputs to differ until you rebuild the registry baseline with `python hash_registry.py`.

For a full validation reference, see `VALIDATION_REPORT.txt` in the root.
