---
title: "Coming from SAS"
vignette: >
  %\VignetteIndexEntry{Coming from SAS}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
format: html
---

```{r}
#| label: setup
#| include: false
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(hvtiRdatabuild)
```

## Who this is for

You know the `tp.bd.*` and `tp.vars.*` templates. You now have to read and
write R. This guide will not teach you R. Its job is to make what you already
know transferable, and to warn you about the handful of places where a
reasonable translation from SAS silently produces a *different, plausible*
number rather than an error.

Read the traps section even if you skip everything else.

## The model shift

A SAS program mutates a dataset through a sequence of steps, and the program
itself is the record of what happened. If an analyst commented out a block with
`%macro skip`, that decision lives only in their copy of the script.

R passes values through functions. The study's `study.yaml` records warehouse
modules and named shared derivation flags. Study-specific predicates and
variable formulas live in committed R code passed to `derive_vars()`. The
`cohort_record` carries upstream selection and later exclusions through the
build, in order. Enabling a shared step is a configuration change; changing a
study-specific rule is a code change.

## Idiom translation

| SAS | R |
|---|---|
| `%include "<dbcreds>.sas"` + `CONNECT TO ODBC` | `dw_connect()` |
| `PROC SQL; SELECT ... FROM connection to ODBC` | `dw_pull(config, conn)` |
| `tp.bd.data.master.sas` | `build_dataset(raw, config, cohort)` |
| `libname library "&STUDY/datasets"` | paths in `study.yaml` |
| `%macro skip; ... %mend skip;` | `modules:` / named shared `derive:` flags in `study.yaml` |
| `%vars(in=built, out=built, transf=1)` | `derive_vars(build, config, study_derive)` |
| `data x; set y; ... run;` | a `dplyr` pipeline |
| `PROC CONTENTS` | `hvtiRutilities::data_dictionary()` |
| `label x = 'Age at surgery';` | `labelled` + `hvtiRutilities::label_map()` |
| `first.id` / `last.id` | `group_by()` + `slice_head()` / `slice_tail()` |
| `libname out xport` then `read.xport()` | parquet, or just return the object |

## Build and derive with study-specific code

This synthetic example shows the boundary between warehouse assembly and local
study rules. The build carries the selected cohort forward. The callback adds a
study variable, records its exclusion stage, and returns only included rows.

```{r}
ids <- c("SYN201", "SYN202", "SYN203")
cohort <- new_cohort_record(
  members = data.frame(
    patient_id = ids, stage_id = "selected", included = TRUE,
    reason = NA_character_
  ),
  stages = data.frame(
    stage_id = "selected", order = 1L, label = "Selected",
    source = "upstream", provenance = "synthetic selection"
  ),
  patient_id_col = "masterid"
)

path <- tempfile(fileext = ".yaml")
yaml::write_yaml(list(
  study = "synthetic",
  cohort_table = "synthetic.cohort",
  warehouse = "synthetic_warehouse",
  view_schema = "dbo",
  pull_date = "2026-10-01",
  modules = list("base", "vitalstatus"),
  derive = list(scale_value = TRUE)
), path)
config <- read_study_config(path)
unlink(path)

base <- data.frame(
  masterid = ids, base_value = c(3, 5, 8), keep = c(TRUE, FALSE, TRUE)
)
status <- data.frame(
  masterid = ids, status_value = c("alive", "alive", "deceased")
)
raw <- structure(list(
  tables = list(bdbase = base, bdstat = status),
  manifest = data.frame(
    module = c("base", "vitalstatus"), output = c("bdbase", "bdstat"),
    n_rows = c(3L, 3L), n_cols = c(3L, 2L),
    pulled_at = as.POSIXct(rep("2026-10-01", 2L), tz = "UTC")
  )
), class = "pull_result")
built <- build_dataset(raw, config, cohort)

study_derive <- function(data, cohort, config) {
  keep <- data$keep
  if (isTRUE(config$derive[["scale_value"]])) {
    data$analysis_value <- data$base_value * 2
  }
  reason <- rep(NA_character_, length(keep))
  reason[!keep] <- "synthetic study exclusion"
  stage <- data.frame(
    stage_id = "study_eligibility",
    order = max(cohort$stages$order) + 1L,
    label = "Study eligibility",
    source = "derive",
    provenance = "synthetic study rule"
  )
  outcomes <- data.frame(
    patient_id = data$masterid, included = keep, reason = reason
  )
  updated <- append_cohort_stage(cohort, stage, outcomes)
  list(data = data[keep, , drop = FALSE], cohort = updated)
}
derived <- derive_vars(built, config, study_derive)
cohort_summary(derived$cohort)$stages
```

Keep study rules in version-controlled R code. Do not put executable R
expressions or source paths in `study.yaml`. Explicitly resolve missing
predicate values before appending outcomes: a missing `included` value is an
error, not an implicit pass or exclusion.

## Port the downstream dataset, not its first consumer

A `bd.SAStoR.sas` job, often copied from `tp.bd.SAStoR.sas`, can look like a
file-format adapter: it reads `built`, keeps a shorter variable list, and writes
`builtr` for R. The useful boundary is not SAS versus R, however. The output is
a downstream data contract that may feed descriptive summaries, models,
tables, and figures long after the first job that happened to read it.

Port that pattern as an analysis set. Declare the columns and any row
exclusions once under `analysis_sets:` in the setup manifest, `_study.yml`,
then let every downstream job that needs that contract read the same
checkpoint. This is not the warehouse-build `study.yaml` read by
`read_study_config()`:

```yaml
analysis_sets:
  primary_analysis:
    id: patient_id
    vars:
      - patient_id
      - surgery_year
      - biomarker
      - event
      - follow_up
    event: event
    expect:
      n: 1200
```

The names and counts above are synthetic. Name a set for the dataset contract,
such as `primary_analysis`, rather than for its first consumer, such as a
particular plot. Add another set only when the study has a different
population or variable contract. If the SAS step keeps every row, omit
`exclude`; the `vars` list alone expresses its projection. `event` names the
set's own event column, which must be one of its `vars`; from it
`write_analysis_set()` calculates `n_events` and `n_censored`, and only a set
with an `event` may add expectations for them.

Write the checkpoint explicitly after the built dataset is ready:

```{r}
#| eval: false
cfg <- hvtiRutilities::study_config()
write_analysis_set("primary_analysis", cfg)
```

Downstream jobs read it; they do not cut their own copies or silently rebuild a
stale one:

```{r}
#| eval: false
d <- read_analysis_set("primary_analysis", cfg)
```

Keep derivation on the build side of this boundary. If the SAS job calls a
`vars` macro before its final `keep`, port those new or transformed variables
to the build or derived-variable layer. An analysis set selects rows and
columns from `built`; it does not create variables. This separation leaves one
reviewable derivation path and one reviewable downstream contract, without a
study-specific package function for either variable list.

## The traps

Each of these produces a wrong number rather than an error. They are the reason
`compare_built()` exists.

### Missing sorts low

In SAS a missing numeric orders *below every number*. So this marks a patient
with unknown BMI as underweight:

```
if bmi < 18.5 then underweight = 1;
```

R does not:

```{r}
bmi <- c(31.2, NA, 17.0)
bmi < 18.5
```

`NA` propagates instead of comparing as small. R is right and SAS is wrong, so
a faithful port will *fail* equivalence here. That is what the
`intentional_divergence` resolution in `equivalence_signoff.yaml` is for. Do
not reproduce the SAS behaviour to make the comparison green.

### Character comparison is blank-padded in SAS

SAS pads character values to their declared length, so `'Smith'` and
`'Smith   '` compare equal. In R they do not:

```{r}
"Smith" == "Smith   "
trimws("Smith   ") == "Smith"
```

This bites hardest on merges keyed by a character id. `compare_built()` trims
before comparing, for exactly this reason.

### SAS has no missing character value

R distinguishes `NA` from `""`. SAS does not: a missing character **is** the
empty string. Read a SAS dataset and every missing character comes back as
`""`:

```{r}
f <- system.file("extdata", "oracle_small.sas7bdat",
                 package = "hvtiRdatabuild")
surgeon <- haven::read_sas(f)$surgeon
surgeon                 # the fourth value was written as NA
is.na(surgeon)          # ... and is not NA any more
```

So a faithful R rebuild holding `NA` disagrees with the oracle holding `""` on
every missing value. `compare_built()` folds `""` to `NA` for character columns
so this does not flood the report with false differences.

The consequence for your own code is sharper: **never test a SAS-sourced
character for `NA`**. `is.na(surgeon)` is `FALSE` even where the value is
missing. Test for `""`, or convert on read.

### `MERGE` is not a join

A SAS data-step `MERGE` on a non-unique BY key does not error. It produces
undefined results. This is not hypothetical. The revision history of
`tp.bd.data.master.sas` records it:

> 07/03/23: Changed the join logic for the FUP dataset to correct many-to-1
> join problem when patient has multiple surgeries in the dataset

R makes the same mistake loud:

```{r}
#| warning: true
#| eval: !expr requireNamespace("dplyr", quietly = TRUE)
a <- data.frame(id = c("A", "A"), x = 1:2)
b <- data.frame(id = c("A", "A"), y = 3:4)
nrow(dplyr::left_join(a, b, by = "id"))
```

Four rows from two, and `dplyr` says so out loud, which SAS never did.

You can silence that warning with `relationship = "many-to-many"`, and there
are joins where the fan-out is genuinely intended. But reach for it only once
you have confirmed the duplication is what you want. Do not pass it reflexively
to quiet the output; that turns off the one check SAS never gave you.

### Dates use different origins

SAS counts days from 1960-01-01; R from 1970-01-01. The gap is 3,653 days.
`haven` converts correctly, so a value read through `snapshot_oracle()` is
already right. A hand-rolled conversion is where this goes wrong.

```{r}
as.Date(0, origin = "1970-01-01")   # R's zero
as.Date(0, origin = "1960-01-01")   # what a raw SAS number means
```

### Automatic variables do not exist in R

`_N_`, `_FREQ_`, and `_TYPE_` are created by SAS. In R you construct them, and
the name is not magic. Phase 0 of this migration found a defect of exactly this
class: `keep _freq_ tau` in one file versus `keep freq tau` in its sibling.
`_FREQ_` is the automatic variable; `freq` is an ordinary one that may not
exist. One of those programs produced wrong counts.

### `length 4` silently truncates

A SAS numeric written with `length 4` loses precision on disk. A value read
back from an old dataset may not equal a freshly computed R value, and **the
SAS side is the lossy one**. `tolerance` in `compare_built()` is what catches
this, but it is an **absolute** threshold, not a relative one. A single
absolute threshold cannot serve columns of mixed magnitude: a value large
enough to be at risk of `length 4` truncation (into the millions, for
identifiers or accumulated counts) can differ by more than a small absolute
`tolerance` while the difference is still storage precision, not a real
disagreement. `1234567` vs `1234567.06` reports `differs`, not
`within_tolerance`, with `max_abs_diff` of `0.06`, but `max_rel_diff` is
`4.9e-08`. For a large-magnitude variable, that tiny relative difference
alongside a `differs` verdict is the signal to suspect storage precision, not
the `within_tolerance` verdict itself. Read `max_rel_diff`, not the verdict
alone, when judging whether a difference is real.

## Verifying your own port

Build the variable in R, then compare against the oracle:

```{r}
oracle <- data.frame(ccfidu = c("A1", "A2", "A3"), age = c(65.5, 70.2, 58.0))
mine   <- data.frame(ccfidu = c("A1", "A2", "A3"), age = c(65.5, 70.2, 58.5))

compare_built(oracle, mine, id = "ccfidu")
```

There is no overall pass or fail, deliberately. Read the table, and record a
resolution for every variable that is not `identical`.