# Normalization Pipeline

This workspace scans and normalizes Quran text workbooks (`*.xlsx`) from the root folder.

## Reproducibility Contract

For deterministic and reproducible results, treat the following as the contract:
- Inputs are the source-of-truth Excel files (`*_Ornamentation.xlsx`) in the workspace root.
- Scripts are executed from the workspace root in the documented order.
- Tool/runtime versions should match the documented environment when possible.

If your runtime differs (Python version, package versions, OS, shell, or any future non-Python implementation/runtime), outputs may differ. The pipeline includes verification scripts so users can confirm reproducibility in their environment.

## Python Version

This workspace is pinned to:
- `Python 3.12.8` (see `.python-version`)

Version note:
- The pipeline can run in other Python versions, but strict reproducibility is guaranteed only when using the pinned version and matching dependencies.

## Setup

From the workspace root:

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install --upgrade pip
pip install -r requirements.txt
```

## Inputs

Place source `.xlsx` files in the workspace root.

These Excel files are the source files for the pipeline.

Expected files currently include:
- `Bazzi_Ornamentation.xlsx`
- `Doori_Ornamentation.xlsx`
- `Hafs_Ornamentation.xlsx`
- `Qaloon_Ornamentation.xlsx`
- `Qumball_Ornamentation.xlsx`
- `Shouba_Ornamentation.xlsx`
- `Soosi_Ornamentation.xlsx`
- `Submission.org_Ornamentation.xlsx`
- `Warsh_Ornamentation.xlsx`

## Reproducible Run

1. Generate Unicode scan reports:

```powershell
python ornamentation_scanner.py --in-dir . --out-dir unicode_reports
```

2. Generate normalized outputs (CSV, JSON, XLSX) and audits/errors:

```powershell
python normalization_engine.py --in-dir . --out-dir normalized_outputs_all_with_json
```

## One-Command Verification

Run a strict reproducibility check (fresh rebuild + naming checks + deterministic text output hashes):

```powershell
.\verify_repro.ps1
```

## Output Naming

Output filenames remove the token `Ornamentation` from source names.

Examples:
- `Hafs_Ornamentation.xlsx` -> `Hafs_normalized.csv`
- `Doori_Ornamentation.xlsx` -> `Doori_unicode_report.json`

## Clean Rebuild (optional)

To regenerate from scratch, remove previous generated folders first:

```powershell
Remove-Item -Recurse -Force .\unicode_reports
Remove-Item -Recurse -Force .\normalized_outputs_all_with_json
```

Then rerun the two commands in `Reproducible Run`.
