---
title: "Dataset Version Tracking with manifest.yaml"
author: "John Ehrlinger"
date: today
format: html
vignette: >
  %\VignetteIndexEntry{Dataset Version Tracking with manifest.yaml}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5
)
```

## The Problem: Datasets Drift

A dataset pulled from Epic today is **not** the same file you will get if you
pull it again tomorrow — even when no one intentionally changed anything.
Backfilled lab results, corrected ICD codes, and updated patient status all
mean that re-running an extraction silently changes the underlying numbers.

The standard defence starts with three habits:

1. **Name every dataset by its extract date**, such as `cohort_20240115.csv`
   or `labs_20240801.sas7bdat`. Never overwrite an existing dated file; a new
   pull gets a new filename.
2. **Record a SHA-256 checksum** the moment a file lands on disk.  A single
   changed byte produces a completely different hash.
3. **Verify the checksums before every analysis run** so that any drift is
   caught immediately, before results are generated.

For an ordinary file manifest, two functions implement those habits:

| Function | Purpose |
|---|---|
| `update_manifest()` | Register a dataset (or update its entry) in `manifest.yaml` |
| `verify_manifest()` | Check every registered dataset before an analysis runs |

A registered study keeps each version of its dataset as a dated file, so a
later rebuild never replaces the one a paper used. We return to that workflow
after the checksum example.

## In a Study: Rebuild, Then Update

Most of the time you do not call any of the functions below directly. A study
registers its dataset once, and from then on the cycle is two steps.

1. Rebuild `datasets/built.sas7bdat` as often as you need. Jobs are not
   affected: they read the registered version, a dated parquet such as
   `built_20260915.parquet`, and `read_built()` says that a newer file is
   waiting.
2. When the rebuild is right, register it:

   ```r
   hvtiRutilities::update_manifest()
   ```

   Run it from anywhere in the study. It converts the new file to
   `built_20261007.parquet`, keeps every earlier version, and prints what it
   did. Commit `manifest.yaml`.

Every registered version stays on disk, so a manuscript revision can be
reproduced from the version its paper used. The rest of this vignette explains
the checksums underneath, and how to record a file outside a study.

## Setup

```{r}
if (requireNamespace("hvtiRutilities", quietly = TRUE)) {
  library("hvtiRutilities")
} else {
  pkgload::load_all(export_all = FALSE, helpers = FALSE, quiet = TRUE)
}
```

For this vignette we work in a temporary directory that mimics a real project's
`datasets/` folder.

```{r}
datasets_dir <- file.path(tempdir(), "datasets")
dir.create(datasets_dir, showWarnings = FALSE)
manifest_path <- file.path(tempdir(), "manifest.yaml")
```

## Registering Datasets

### CSV — automatic row count

Generate a synthetic cohort (using `generate_survival_data()`) and write it as
a date-stamped CSV.

```{r}
set.seed(42)
cohort <- generate_survival_data(n = 200, seed = 42)

cohort_file <- file.path(datasets_dir, "cohort_20240115.csv")
write.csv(cohort, cohort_file, row.names = FALSE)
```

Register it in the manifest.  Row count is detected automatically from the CSV
header; only the SHA-256 is computed from the raw bytes.

```{r}
update_manifest(
  file         = cohort_file,
  manifest_path = manifest_path,
  extract_date = "2024-01-15",
  source       = "Epic EMR, query v4.2, ICD mapping v3.2",
  sort_key     = "ccfid"
)
```

Add a second file — a simulated labs extract pulled on the same date.

```{r}
set.seed(7)
labs <- data.frame(
  ccfid   = cohort$ccfid,
  hgb     = round(rnorm(200, mean = 13.5, sd = 1.8), 1),
  egfr    = round(rnorm(200, mean = 72,   sd = 18),   0),
  bnp     = round(exp(rnorm(200, mean = 4.8, sd = 0.9)), 0)
)

labs_file <- file.path(datasets_dir, "labs_20240115.csv")
write.csv(labs, labs_file, row.names = FALSE)

update_manifest(
  file          = labs_file,
  manifest_path = manifest_path,
  extract_date  = "2024-01-15",
  source        = "Epic EMR, labs module — Hgb, eGFR, BNP",
  sort_key      = "ccfid"
)
```

### Inspecting the manifest

The manifest is a plain YAML file committed to version control alongside your
analysis scripts (but **not** alongside the data files, which stay out of git).

```{r}
cat(paste(readLines(manifest_path), collapse = "\n"))
```

Each entry records exactly what is needed to re-verify the file later:
`extract_date`, `n_rows`, `sha256`, and any provenance notes you supply.

### SAS datasets — automatic row count

Files exported directly from SAS (`.sas7bdat`) are handled the same way.
`update_manifest()` calls `haven::read_sas()` internally to count rows;
the checksum is always computed from the raw file bytes.

```{r eval=FALSE}
update_manifest(
  file          = here::here("datasets", "cohort_20240115.sas7bdat"),
  manifest_path = here::here("manifest.yaml"),
  extract_date  = "2024-01-15",
  source        = "SAS CORR registry extract, query v3.1",
  sort_key      = "pat_id"
)
```

### Excel workbooks — automatic row count

Adjudication logs, crosswalk tables, and other curator-maintained datasets
often arrive as `.xlsx` files.  `update_manifest()` calls
`readxl::read_excel()` to count rows.

```{r eval=FALSE}
update_manifest(
  file          = here::here("datasets", "adjudication_20240115.xlsx"),
  manifest_path = here::here("manifest.yaml"),
  extract_date  = "2024-01-15",
  source        = "Clinical Events Committee adjudication log, v2"
)
```

### Other formats — supply `n_rows` explicitly

For RDS, Feather, Parquet, or any format not listed above, pass `n_rows`
directly.  The SHA-256 is still computed from the raw bytes.

```{r eval=FALSE}
update_manifest(
  file          = here::here("datasets", "cohort_20240115.rds"),
  manifest_path = here::here("manifest.yaml"),
  extract_date  = "2024-01-15",
  n_rows        = nrow(readRDS(here::here("datasets", "cohort_20240115.rds"))),
  source        = "RDS cached from Epic pull"
)
```

## Verifying Before Every Analysis

Place `verify_manifest()` at the very top of every analysis script and Quarto
document — before any data are loaded and before any results are produced.

```{r}
verify_manifest(
  manifest_path = manifest_path,
  data_dir      = datasets_dir
)
```

The function prints one confirmation line per file and returns invisibly.  If
everything is clean, your script continues.

### Typical script header

```{r eval=FALSE}
library(hvtiRutilities)
library(here)

# Abort immediately if any registered dataset has changed
verify_manifest(here("manifest.yaml"))

# Only reached if all checksums pass
cohort <- read.csv(here("datasets", "cohort_20240115.csv"))
labs   <- read.csv(here("datasets", "labs_20240115.csv"))
```

## What Happens When Data Changes

Suppose the data warehouse silently updates `cohort_20240115.csv` overnight.
`verify_manifest()` catches it the moment the next analysis run starts.

```{r error=TRUE}
# Simulate a silent dataset change
cohort_modified <- cohort
cohort_modified$age[1] <- cohort_modified$age[1] + 1L
write.csv(cohort_modified, cohort_file, row.names = FALSE)

# verify_manifest stops immediately
verify_manifest(
  manifest_path = manifest_path,
  data_dir      = datasets_dir
)
```

The error message names the affected file and shows both the expected and
actual SHA-256 values, giving you an unambiguous audit trail.

### Collecting all failures at once

During development it can be useful to see every problem before fixing any of
them.  Set `stop_on_error = FALSE` to collect warnings instead of stopping.

```{r warning=TRUE}
report <- verify_manifest(
  manifest_path = manifest_path,
  data_dir      = datasets_dir,
  stop_on_error = FALSE
)

report[report$status == "FAIL", c("file", "message")]
```

### Correcting a file

A changed checksum tells you that the bytes moved. It does not tell you which
bytes were correct, and an already overwritten file cannot be reconstructed
from its checksum. Keep the mutable work under a working name while the
statistical programmer builds and validates it, and register it as a dated
file only when it is right. Registering is the boundary: a dated file is not
rewritten afterwards.

The rest of this vignette still needs its original temporary file. This reset
is safe only because the fixture is disposable; it is not a recovery method
for a study whose registered file was overwritten.

```{r}
write.csv(cohort, cohort_file, row.names = FALSE)
```

## Looking at a Rebuild Before Registering It

After a rebuild, jobs keep reading the registered version. Before you register
the new file, `review_built()` shows how it differs from the version jobs read.
It reads both files and writes nothing.

```{r eval=FALSE}
review_built(study_config())
```

When the working file is unchanged since the last registration, it says so.
When it differs, it prints the comparison of rows, columns, types and labels
and points to `update_manifest()`, which registers the file as the next
version.

```{r eval=FALSE}
hvtiRutilities::update_manifest()
```

`review_built()` describes how the data changed. It does not certify that the
new file is correct; that judgement is yours.

## Running a Job on the Working File

Sometimes a job needs the rebuilt data before it has a dated version, for
example a descriptive report you want to look at while the build is still being
checked. `read_built(working = TRUE)` reads the working file, and so does
setting `HVTI_DATA=working` for the render.

```{sh eval=FALSE}
HVTI_DATA=working quarto render 10_descriptive/table1.qmd
```

Any read that is not of a dated version is a working read, whether or not you
asked for one. It is allowed in the interactive console and in jobs in the
study's `datasets` and `descriptive` folders. In distribution, analysis and
graph jobs it stops with a message that says to register the dataset or unset
`HVTI_DATA`. A working read signals `hvtiRutilities_working_read`, a message
that says the results are a draft. Treat the report that comes out as a draft
too.

A study still on a legacy registration, one with no dated version, has only
its working file to read. Its analysis jobs stop until `update_manifest()`
converts it to a dated version.

## How a Version Was Produced

Each registered version records where its data came from, under a
`provenance:` map in `manifest.yaml` keyed by the parquet name.

* `origin: bd` means a build left a build record, `<stem>.build.yml`, beside
  the working file, and that record describes exactly the file registered.
  `update_manifest()` moves it next to the dated parquet and the manifest
  notes its name and checksum, with the job, view and as-of date it carries.
* `origin: legacy` means there was no such record. Every version registered
  before this scheme reads as legacy.

`verify_manifest()` checks each build record against its recorded checksum.
The record is a file beside the dataset on the share. It is not committed:
only `manifest.yaml` goes into Git.

## A New Pull for a Different Project

When the same registry is pulled again months later for a different analysis,
it gets a **new filename** — never overwriting the original.

```{r}
set.seed(99)
cohort_aug <- generate_survival_data(n = 315, seed = 99)

cohort_aug_file <- file.path(datasets_dir, "cohort_20240801.csv")
write.csv(cohort_aug, cohort_aug_file, row.names = FALSE)

update_manifest(
  file          = cohort_aug_file,
  manifest_path = manifest_path,
  extract_date  = "2024-08-01",
  source        = "Epic EMR, query v4.2, ICD mapping v3.3 — LVAD sub-study cohort",
  sort_key      = "ccfid"
)
```

The manifest now tracks all three files independently.

```{r}
m <- yaml::read_yaml(manifest_path)
do.call(rbind, lapply(m$datasets, function(e) {
  data.frame(file = e$file, extract_date = e$extract_date,
             n_rows = e$n_rows, stringsAsFactors = FALSE)
}))
```

Verification covers all of them in a single call.

```{r}
verify_manifest(
  manifest_path = manifest_path,
  data_dir      = datasets_dir
)
```

## What Changed Inside the File?

The manifest answers one question, and it answers it absolutely: did this file
change? A single altered byte flips the checksum. But a failed `verify_manifest()`
tells you nothing about *what* moved, and that is the question you actually have
to answer before you decide whether the change is benign.

`compare_datasets()` answers it. Give it the two versions and it reports the
four kinds of drift that matter:

```{r}
cohort_v1 <- generate_survival_data(n = 200, seed = 42)
cohort_v2 <- generate_survival_data(n = 315, seed = 99)

# Changes a real re-pull might bring
cohort_v2$creatinine <- 1.1
cohort_v2$reop <- NULL
cohort_v2$dead <- as.integer(cohort_v2$dead) > 0
labelled::var_label(cohort_v2$gfr_bs) <- "eGFR at discharge (mL/min/1.73m2)"

compare_datasets(cohort_v1, cohort_v2)
```

Read that output in order of how much trouble each line will cause you.

Added and dropped columns are the easy ones; a model referencing `reop` will
fail loudly on the next run. The type change on `dead` is worse, because
`integer` to `logical` is a change most code will accept silently. Notice it
also cost the column its label, which is how a type change usually announces
itself in the audit.

The label change on `gfr_bs` is the one to watch. Baseline eGFR and eGFR at
discharge are different measurements under the same column name, and nothing
but the label will tell you they were swapped. Checksums catch it as "changed";
only the label comparison tells you the variable now means something else.

The result is a list, so a script can act on it rather than print it:

```{r}
drift <- compare_datasets(cohort_v1, cohort_v2)
drift$cols_dropped
drift$label_changes
```

Use the two tools together: `verify_manifest()` at the top of the script to
stop the run, then `compare_datasets()` to work out why. See
[PROC CONTENTS and PROC MEANS in R](sas-procedures.html) for more on inspecting
an extract once you know it moved.

## Policy Recommendations

| Stage | Action |
|---|---|
| **While rebuilding** | Work on the working file; jobs keep reading the registered version |
| **When the rebuild looks right** | Run `review_built()`, then `update_manifest()`; commit `manifest.yaml` |
| **Top of every analysis script** | Call `verify_manifest()` — before loading any data |
| **At manuscript submission** | Tag the git commit; freeze `manifest.yaml`; do not re-pull data |
| **For an unregistered legacy file** | Use `update_manifest()` to record its current bytes; a checksum cannot recover an overwritten version |
| **`.gitignore`** | Add `datasets/` — commit the manifest, not the data |

## Session Information

```{r}
sessionInfo()
```