The README's Installation section now leads with hvtiR::install(), which
installs the whole HVTI R family in one call, and then gives
pak::pak("ehrlinger/hvtiRdatabuild") for this package alone.
Tests and examples pass renv and cleanup to hvtiRutilities::study_setup() explicitly, ahead of
hvtiRutilities turning both on by default. A new test fails on any call that leaves them implicit.
The hvtiRutilities floor rises to 1.5.2, the first release whose study_setup() takes renv and cleanup.
Now requires R 4.4.0 or newer, up from 4.1.0, to match the rest of the
HVTI family. hvtiR::install() installs the members together, and several
already required 4.4.0, so on an older R the install failed whatever this
package declared. Now requires hvtiRutilities 1.5.1.
The build layer. Three pieces sit between dw_pull() and the analysis
sets. build_dataset() joins a study's patient-level modules on its cohort
and keeps repeated-event tables beside the result rather than widening them.
derive_vars() runs the study's own R code, committed with the study, and
takes back the data with an updated cohort. A cohort_record, from
new_cohort_record(), append_cohort_stage() and cohort_summary(), carries
inclusion, exclusion and attrition history through both, in order. A missing
inclusion value is an error, not an implicit pass or exclusion. Duplicate
source identifier columns are kept for audit, patient identifiers stay stable
across joins, and the print() and str() methods and callback diagnostics
never show a patient value. vignette("coming-from-sas") walks the build,
study code and attrition through a synthetic example. dplyr moves from
Suggests to Imports.
A stale analysis set no longer stops a draft. read_analysis_set()
reads a set whose parent or declaration has changed, and signals
hvtiRutilities_stale_analysis_set with the write_analysis_set() call that
updates it. A final render (HVTI_TEMPLATE_STRICT set, as
render_job(final = TRUE) does) still stops, with the same commands. This
reverses the earlier rule that every stale read stopped; a damaged set still
stops in every render.
An analysis set's parent is now the registered version of the built dataset
when hvtiRutilities registered it as a dated parquet. Rebuilding
built.sas7bdat without registering it no longer makes every set stale,
because jobs read nothing new until hvtiRutilities::update_manifest()
registers it. Sets written before this change name the SAS file as their
parent, so each is stale once after the study's first update_manifest();
write_analysis_set() refreshes it. A dataset still registered the old way
is identified by its source, as before.
write_analysis_set() and read_analysis_set() skip the line of text that
hvtiRutilities 1.5.1 writes at the head of a manifest.yaml holding a
registered dated version. Without this they stopped with "$ operator is
invalid for atomic vectors" on every study registered from 1.5.1 on.
write_analysis_set() refuses a set named for a registered dated version,
such as built_20261008, before writing anything. Its parquet would have
replaced that version, the registered data every job reads.
DESCRIPTION now declares the Quarto command line tool in
SystemRequirements. The vignettes have always needed it to build; the
field makes that visible to installers and to R CMD check.
Articles on the pkgdown site put the table of contents on the left and use
the full width of the window, through pkgdown/extra.css. The installed
vignettes are unchanged: the Quarto vignette engine renders them in its own
minimal format, which has no sidebar layout.
Master datasets. Six new functions snapshot, lift and correct the master
datasets that study builds read. read_master_config() reads a master.yml
declaring a master's key, alternate keys and parent. snapshot_master()
freezes its SAS builds as parquet and records which parent release each was
built from, read from the run's log first. lift_master() loads a snapshot
into the warehouse behind key and full-parity gates and creates the master's
view. backfill_corrections(), propose_correction() and
decide_correction() keep an append-only record of corrections the view
applies. The bulk functions dry-run by default. See
vignette("master-datasets"). duckdb and tidyselect join Suggests.
An analysis set names its own event column. hvtiRutilities 1.4.0 made
study registration endpoint-neutral: register_data() no longer takes
event or time, and study_config() no longer carries a study-wide
cohort. write_analysis_set() had counted n_events and n_censored from
that cohort, so under 1.4.0 every write failed. A set now declares event:
(one of its vars) to get both counts, and an expect on either count
without one stops with a message naming the missing key. A set with no
event records n alone. A set that relied on the study-wide cohort needs
the one-line event: added, which also changes its declaration hash, so
read_analysis_set() asks for one rewrite.
hvtiRdatabuild now requires hvtiRutilities 1.4.0 or later, since its tests
and registration fixture use the endpoint-neutral register_data().
snapshot_oracle() can snapshot a dataset too large for memory. The new
chunk_rows argument reads the SAS dataset a chunk at a time and writes the
chunks as row groups of one parquet file. It also records a SHA-256 of the
SAS source beside the parquet checksum, and writes a .meta.json sidecar
holding each column's label, SAS format and type, so that metadata survives
into systems that cannot read R attributes. jsonlite joins Suggests. The
source checksum now brackets the read, taken before and after; a mismatch
removes the parquet and stops rather than write a snapshot of a file that
changed underneath it.
New publish_dataset(). A programmer can publish a mutable clinical-data
draft as an immutable dated release. Publication preserves the draft bytes,
verifies that the staged copy is readable, records its checksum and shape in
dataset-catalog.yml, and serializes concurrent publishers with a catalog
lock. Same-day corrections receive revision filenames; retrying identical
bytes is idempotent, including recovery of a file moved before a failed
catalog write.
New withdraw_dataset_release(). Withdrawal records a reason and an
optional replacement in the catalog without changing or deleting the
published file. The package now requires hvtiRutilities 1.3.1, whose study
workflow discovers, reviews, and explicitly adopts these releases.
Documentation no longer points dw_pull() users at a
building-a-study-dataset vignette that does not exist, and the SAS migration
guide no longer lists the shipped dw_connect() and dw_pull() as future
work. Prose across the README, vignette and reference pages was tightened.
The SAS migration guide now treats bd.SAStoR.sas outputs as general
downstream data contracts rather than exports owned by their first consumer.
It shows how the final selection becomes a shared analysis set while variable
derivation remains in the build layer.
Analysis-set declarations now reject duplicate variables and require every expected row/event count to be one non-negative whole number. This prevents duplicate parquet columns from being silently renamed and fractional counts from matching after integer truncation.
Analysis sets resolve their outputs through the study's logical datasets directory, preserving numbered new studies and adopted legacy layouts.
write_analysis_set() and read_analysis_set(). An analysis set
is a declared, checkpointed selection of the built dataset: the columns a
job reads and the rows it excludes, declared under analysis_sets: in
_study.yml, written once to datasets/ as <name>.parquet with a sidecar
recording its parent and per-rule attrition, and read by every job that
needs it. Reading stops, rather than rebuilding, when the built dataset or
the declaration has changed.hvtiRdatasets to hvtiRdatabuild. The package exports six
functions and no data object, so a ...datasets name promised a payload and delivered a
pipeline. Update library() calls and any hvtiRdatasets:: prefixes. The repository
moved to github.com/ehrlinger/hvtiRdatabuild; GitHub redirects the old URL, so an
existing remotes::install_github("ehrlinger/hvtiRdatasets") keeps resolving.hvtiRdatasets.show_ids is now hvtiRdatabuild.show_ids. There is no
fallback to the old name: a stale options(hvtiRdatasets.show_ids = TRUE) is ignored and
identifiers are not printed. That is the conservative direction for a flag whose own
documentation warns it may emit PHI.No function changed behaviour, so results are identical to 0.1.2.
Adds this changelog. No function changed, so results are identical to 0.1.1.
NEWS.md covers 0.1.0 and 0.1.1 retroactively, reconstructed from the commit
history. It ships with the package, so utils::news(package = "hvtiRdatabuild")
reads it and the pkgdown site gains a Changelog.AGENTS.md now says which changes bump the version and which do not. The rule
was unconditional, but the docs-only commits on main had never bumped
anything, and a rule the repository visibly does not follow teaches the wrong
lesson. Files excluded by .Rbuildignore do not ship, so they do not bump.Slice S1 (Pull). The package can now read a study's configuration, open a credentialed warehouse connection, and pull the modules the study declares. Nothing in the S0 verify path changed, so a comparison run against 0.1.0 returns the same verdicts.
read_study_config() reads a study YAML into a validated configuration. An
unknown key is an error rather than a silent ignore, and modules must be a
sequence: a YAML mapping there is rejected.dw_connect() opens the warehouse connection and reports which credential
source it resolved. The ladder stops at the first one available: Kerberos
integrated authentication, then a named ODBC DSN, then HVI_DW_UID and
HVI_DW_PWD from ~/.Renviron, then keyring. A DSN outranks .Renviron
because the driver reads a DSN password and it never enters R's memory, where
Sys.getenv() prints it and a handler dumping the environment on error
captures it. A credential file with a loose mode is an error, not a warning,
and a project-level .Renviron shadowing the user-level one warns rather
than failing somewhere further downstream.dw_modules() returns the available warehouse modules as a data frame of
module, output, join_key, and optional. The definitions ship as YAML
under inst/extdata/modules/, not as R code, so the SQL can carry
placeholders and the real warehouse, schema, and cohort names arrive from
study.yaml at run time. That is what keeps site identifiers out of a public
repository.dw_pull() pulls the modules a study declares and returns the raw tables
with a manifest of module, output, n_rows, n_cols, and pulled_at.
It is read-only, and a test asserts that no DBI write entry point is ever
called. Colliding module output names error before any query runs, and an
empty required module is an error.print.pull_result() prints the manifest shape only, never row-level data,
so there are no identifiers to redact.compare_built() measures bdbase and bdstat only. It requires one row
per identifier on both sides, and echo, fup, and bdevents are one row
per event by construction (echocardiogram, follow-up visit, reoperation), so
a patid legitimately repeats and the comparison errors on them.
Composite-key comparison is the eventual fix, deferred to a later slice.libsql) is not ported. The
re-pull variants that need it, snapshotpull and ccfpull, arrive with it.man/
does not drift against whatever the runner happens to install.R CMD check across platforms,
the PDF manual build, lintr, the pkgdown site build, and coverage upload.
DESCRIPTION now lists the pkgdown URL.First release. Slice S0 (Verify): freeze a SAS-built dataset as a checksummed reference, then compare an R rebuild against it.
snapshot_oracle() reads a SAS dataset once with haven, writes parquet,
and records a SHA-256. A SAS dataset on a shared volume can be regenerated at
any time, and when it is, every comparison that passed against it silently
stops meaning anything. Compare against the snapshot, and cite the checksum.compare_built() joins an R build to its oracle on an identifier and
classifies every variable as identical, within_tolerance, differs,
absent_in_r, absent_in_sas, or type_mismatch.print.built_comparison() and str.built_comparison() summarise a
comparison without printing identifiers, which may be PHI.compare_built() returns a table and never an overall pass/fail. Several
hundred variables collapsed into one boolean turns a real difference into a
green check. Read the table.if bmi < 18.5 classifies a missing BMI as underweight where
R gives NA. Record those cases in equivalence_signoff.yaml as
intentional_divergence.haven out of the chain of custody, it confines
it to one audited step. A misread is faithfully preserved in the parquet, so
supply expect to validate the conversion against SAS-side PROC CONTENTS
output rather than treating a clean round-trip as a correct read.Test fixtures are invented and say so. .gitignore blocks *.sas7bdat and
*.parquet, with one deliberate exception for the synthetic
inst/extdata/oracle_small.sas7bdat. The integration test against a real study
runs only when HVTI_ORACLE_DIR points at real SAS datasets, a directory that
lives outside this repository, and its assertions are on shape and verdicts
only, so no patient value can reach a failure message.