# Normalization Pipeline

This workspace scans ornamented Quran Excel workbooks and produces reproducible normalized outputs.

## Reproducibility Contract

For Phase 1 reproducibility, the contract is:

- Source Excel workbooks (`*.xlsx`) are the source of truth.
- Scripts are run from the workspace root.
- Excel files may be placed in the root or subfolders; the scripts discover them recursively.
- Generated outputs are rebuilt from source files, not manually edited.
- Strict reproducibility is tied to the pinned Python/runtime environment.

## Python Version

Pinned runtime:

```text
Python 3.12.8
```

See `.python-version`.

## Setup

From the workspace root:

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install --upgrade pip
pip install -r requirements.txt
```

## Inputs

Place ornamentation Excel workbooks anywhere inside the workspace.

Expected workbook shape:

```text
Column A = sura
Column B = verse
Column C = text
```

Current completed 7-reader transmission set:

- Hafs
- Shouba
- Warsh
- Qaloon
- Bazzi
- Qumball
- Doori
- Soosi
- Hishem
- Ibn Dhakwan
- Khalaf Hamza
- Khallad
- Kisai_Duri
- Kisai_Layth
- Abu al-Harith

Verse-system variants may include:

- Basra
- Damascus
- Himsi
- Kufa
- Mecca
- Medina I
- Medina II
- VERSE 0

## Reproducible Run

Generate Unicode scan reports:

```powershell
python ornamentation_scanner.py --in-dir . --out-dir unicode_reports
```

Generate normalized outputs:

```powershell
python normalization_engine.py --in-dir . --out-dir normalized_outputs_all_with_json
```

Generate ornamentation JSON exports:

```powershell
python build_ornamentation_json.py --in-dir . --out-dir ornamentation_json
```

Optional strict normalization mode:

```powershell
python normalization_engine.py --in-dir . --out-dir normalized_outputs_all_with_json --fail-fast
```

## Verify Reproducibility

Run the reproducibility verifier:

```powershell
.\verify_repro.ps1
```

This verifies a fresh rebuild of the scanner/normalization pipeline and checks deterministic generated text-output hashes across repeated runs.

## Output Layout

```text
unicode_reports/
normalized_outputs_all_with_json/
  csv/
  excel/
  json/
  _audit/
  _errors/
ornamentation_json/
```

No TXT normalization output is produced by `normalization_engine.py`.

## Output Naming

Output filenames remove the token `Ornamentation` from source names and retain verse-system suffixes.

Example:

```text
Hafs_Ornamentation(Medina I).xlsx
Hafs(Medina I)_normalized.csv
```

## Hash Registry

Generate a workspace hash registry:

```powershell
.\hash_registry.ps1 -Mode generate
```

Validate the workspace against the registry:

```powershell
.\hash_registry.ps1 -Mode validate
```

To include generated outputs in the registry:

```powershell
.\hash_registry.ps1 -Mode generate -IncludeGeneratedOutputs
```

Use the hash registry only after the scripts, README, and generated outputs are in the intended final Phase 1 state.

## Reference Docs

- `NORMALIZATION_RULES.txt`
- `PIPELINE_VALIDATION.txt`
