Changes in version 0.3.2 - The README's Installation section now leads with hvtiR::install(), which installs the whole HVTI R family in one call, and then gives pak::pak("ehrlinger/hvtiRdatabuild") for this package alone. - Tests and examples pass renv and cleanup to hvtiRutilities::study_setup() explicitly, ahead of hvtiRutilities turning both on by default. A new test fails on any call that leaves them implicit. - The hvtiRutilities floor rises to 1.5.2, the first release whose study_setup() takes renv and cleanup. Changes in version 0.3.1 - The test of an analysis set over a dataset registered before dated versions now writes the set as a datasets job would. hvtiRutilities' stage gate serves a dataset with no dated version only to the console and to datasets or descriptive jobs, so the test's non-interactive run was refused. Test-only. Changes in version 0.3.0 - Now requires R 4.4.0 or newer, up from 4.1.0, to match the rest of the HVTI family. hvtiR::install() installs the members together, and several already required 4.4.0, so on an older R the install failed whatever this package declared. Now requires hvtiRutilities 1.5.1. - The build layer. Three pieces sit between dw_pull() and the analysis sets. build_dataset() joins a study's patient-level modules on its cohort and keeps repeated-event tables beside the result rather than widening them. derive_vars() runs the study's own R code, committed with the study, and takes back the data with an updated cohort. A cohort_record, from new_cohort_record(), append_cohort_stage() and cohort_summary(), carries inclusion, exclusion and attrition history through both, in order. A missing inclusion value is an error, not an implicit pass or exclusion. Duplicate source identifier columns are kept for audit, patient identifiers stay stable across joins, and the print() and str() methods and callback diagnostics never show a patient value. vignette("coming-from-sas") walks the build, study code and attrition through a synthetic example. dplyr moves from Suggests to Imports. - A stale analysis set no longer stops a draft. read_analysis_set() reads a set whose parent or declaration has changed, and signals hvtiRutilities_stale_analysis_set with the write_analysis_set() call that updates it. A final render (HVTI_TEMPLATE_STRICT set, as render_job(final = TRUE) does) still stops, with the same commands. This reverses the earlier rule that every stale read stopped; a damaged set still stops in every render. - An analysis set's parent is now the registered version of the built dataset when hvtiRutilities registered it as a dated parquet. Rebuilding built.sas7bdat without registering it no longer makes every set stale, because jobs read nothing new until hvtiRutilities::update_manifest() registers it. Sets written before this change name the SAS file as their parent, so each is stale once after the study's first update_manifest(); write_analysis_set() refreshes it. A dataset still registered the old way is identified by its source, as before. - write_analysis_set() and read_analysis_set() skip the line of text that hvtiRutilities 1.5.1 writes at the head of a manifest.yaml holding a registered dated version. Without this they stopped with "$ operator is invalid for atomic vectors" on every study registered from 1.5.1 on. - write_analysis_set() refuses a set named for a registered dated version, such as built_20261008, before writing anything. Its parquet would have replaced that version, the registered data every job reads. - DESCRIPTION now declares the Quarto command line tool in SystemRequirements. The vignettes have always needed it to build; the field makes that visible to installers and to R CMD check. - Articles on the pkgdown site put the table of contents on the left and use the full width of the window, through pkgdown/extra.css. The installed vignettes are unchanged: the Quarto vignette engine renders them in its own minimal format, which has no sidebar layout. Changes in version 0.2.4 - Master datasets. Six new functions snapshot, lift and correct the master datasets that study builds read. read_master_config() reads a master.yml declaring a master's key, alternate keys and parent. snapshot_master() freezes its SAS builds as parquet and records which parent release each was built from, read from the run's log first. lift_master() loads a snapshot into the warehouse behind key and full-parity gates and creates the master's view. backfill_corrections(), propose_correction() and decide_correction() keep an append-only record of corrections the view applies. The bulk functions dry-run by default. See vignette("master-datasets"). duckdb and tidyselect join Suggests. - An analysis set names its own event column. hvtiRutilities 1.4.0 made study registration endpoint-neutral: register_data() no longer takes event or time, and study_config() no longer carries a study-wide cohort. write_analysis_set() had counted n_events and n_censored from that cohort, so under 1.4.0 every write failed. A set now declares event: (one of its vars) to get both counts, and an expect on either count without one stops with a message naming the missing key. A set with no event records n alone. A set that relied on the study-wide cohort needs the one-line event: added, which also changes its declaration hash, so read_analysis_set() asks for one rewrite. hvtiRdatabuild now requires hvtiRutilities 1.4.0 or later, since its tests and registration fixture use the endpoint-neutral register_data(). - snapshot_oracle() can snapshot a dataset too large for memory. The new chunk_rows argument reads the SAS dataset a chunk at a time and writes the chunks as row groups of one parquet file. It also records a SHA-256 of the SAS source beside the parquet checksum, and writes a .meta.json sidecar holding each column's label, SAS format and type, so that metadata survives into systems that cannot read R attributes. jsonlite joins Suggests. The source checksum now brackets the read, taken before and after; a mismatch removes the parquet and stops rather than write a snapshot of a file that changed underneath it. Changes in version 0.2.3 - New publish_dataset(). A programmer can publish a mutable clinical-data draft as an immutable dated release. Publication preserves the draft bytes, verifies that the staged copy is readable, records its checksum and shape in dataset-catalog.yml, and serializes concurrent publishers with a catalog lock. Same-day corrections receive revision filenames; retrying identical bytes is idempotent, including recovery of a file moved before a failed catalog write. - New withdraw_dataset_release(). Withdrawal records a reason and an optional replacement in the catalog without changing or deleting the published file. The package now requires hvtiRutilities 1.3.1, whose study workflow discovers, reviews, and explicitly adopts these releases. Changes in version 0.2.2 - Documentation no longer points dw_pull() users at a building-a-study-dataset vignette that does not exist, and the SAS migration guide no longer lists the shipped dw_connect() and dw_pull() as future work. Prose across the README, vignette and reference pages was tightened. - The SAS migration guide now treats bd.SAStoR.sas outputs as general downstream data contracts rather than exports owned by their first consumer. It shows how the final selection becomes a shared analysis set while variable derivation remains in the build layer. - Analysis-set declarations now reject duplicate variables and require every expected row/event count to be one non-negative whole number. This prevents duplicate parquet columns from being silently renamed and fractional counts from matching after integer truncation. - Analysis sets resolve their outputs through the study's logical datasets directory, preserving numbered new studies and adopted legacy layouts. Changes in version 0.2.1 - New write_analysis_set() and read_analysis_set(). An analysis set is a declared, checkpointed selection of the built dataset: the columns a job reads and the rows it excludes, declared under analysis_sets: in _study.yml, written once to datasets/ as .parquet with a sidecar recording its parent and per-rule attrition, and read by every job that needs it. Reading stops, rather than rebuilding, when the built dataset or the declaration has changed. Changes in version 0.2.0 Breaking Changes - Package renamed from hvtiRdatasets to hvtiRdatabuild. The package exports six functions and no data object, so a ...datasets name promised a payload and delivered a pipeline. Update library() calls and any hvtiRdatasets:: prefixes. The repository moved to github.com/ehrlinger/hvtiRdatabuild; GitHub redirects the old URL, so an existing remotes::install_github("ehrlinger/hvtiRdatasets") keeps resolving. - The print option hvtiRdatasets.show_ids is now hvtiRdatabuild.show_ids. There is no fallback to the old name: a stale options(hvtiRdatasets.show_ids = TRUE) is ignored and identifiers are not printed. That is the conservative direction for a flag whose own documentation warns it may emit PHI. No function changed behaviour, so results are identical to 0.1.2. Changes in version 0.1.2 Adds this changelog. No function changed, so results are identical to 0.1.1. - NEWS.md covers 0.1.0 and 0.1.1 retroactively, reconstructed from the commit history. It ships with the package, so utils::news(package = "hvtiRdatabuild") reads it and the pkgdown site gains a Changelog. - AGENTS.md now says which changes bump the version and which do not. The rule was unconditional, but the docs-only commits on main had never bumped anything, and a rule the repository visibly does not follow teaches the wrong lesson. Files excluded by .Rbuildignore do not ship, so they do not bump. Changes in version 0.1.1 Slice S1 (Pull). The package can now read a study's configuration, open a credentialed warehouse connection, and pull the modules the study declares. Nothing in the S0 verify path changed, so a comparison run against 0.1.0 returns the same verdicts. New features - read_study_config() reads a study YAML into a validated configuration. An unknown key is an error rather than a silent ignore, and modules must be a sequence: a YAML mapping there is rejected. - dw_connect() opens the warehouse connection and reports which credential source it resolved. The ladder stops at the first one available: Kerberos integrated authentication, then a named ODBC DSN, then HVI_DW_UID and HVI_DW_PWD from ~/.Renviron, then keyring. A DSN outranks .Renviron because the driver reads a DSN password and it never enters R's memory, where Sys.getenv() prints it and a handler dumping the environment on error captures it. A credential file with a loose mode is an error, not a warning, and a project-level .Renviron shadowing the user-level one warns rather than failing somewhere further downstream. - dw_modules() returns the available warehouse modules as a data frame of module, output, join_key, and optional. The definitions ship as YAML under inst/extdata/modules/, not as R code, so the SQL can carry placeholders and the real warehouse, schema, and cohort names arrive from study.yaml at run time. That is what keeps site identifiers out of a public repository. - dw_pull() pulls the modules a study declares and returns the raw tables with a manifest of module, output, n_rows, n_cols, and pulled_at. It is read-only, and a test asserts that no DBI write entry point is ever called. Colliding module output names error before any query runs, and an empty required module is an error. - print.pull_result() prints the manifest shape only, never row-level data, so there are no identifiers to redact. Scope limits worth knowing - compare_built() measures bdbase and bdstat only. It requires one row per identifier on both sides, and echo, fup, and bdevents are one row per event by construction (echocardiogram, follow-up visit, reoperation), so a patid legitimately repeats and the comparison errors on them. Composite-key comparison is the eventual fix, deferred to a later slice. - The cohort write-back the SAS templates perform (libsql) is not ported. The re-pull variants that need it, snapshotpull and ccfpull, arrive with it. Documentation and infrastructure - Roxygen markdown is enabled, so cross-references in the roxygen blocks resolve. - Regenerated with roxygen2 8.1.0, with CI pinned to that version so man/ does not drift against whatever the runner happens to install. - The repository is on the house CI standard: R CMD check across platforms, the PDF manual build, lintr, the pkgdown site build, and coverage upload. DESCRIPTION now lists the pkgdown URL. Changes in version 0.1.0 First release. Slice S0 (Verify): freeze a SAS-built dataset as a checksummed reference, then compare an R rebuild against it. New features - snapshot_oracle() reads a SAS dataset once with haven, writes parquet, and records a SHA-256. A SAS dataset on a shared volume can be regenerated at any time, and when it is, every comparison that passed against it silently stops meaning anything. Compare against the snapshot, and cite the checksum. - compare_built() joins an R build to its oracle on an identifier and classifies every variable as identical, within_tolerance, differs, absent_in_r, absent_in_sas, or type_mismatch. - print.built_comparison() and str.built_comparison() summarise a comparison without printing identifiers, which may be PHI. Design notes - compare_built() returns a table and never an overall pass/fail. Several hundred variables collapsed into one boolean turns a real difference into a green check. Read the table. - Row-set differences are reported apart from value differences. An identifier present on one side only is a cohort discrepancy, a different class of problem from a miscomputed variable, and reporting the two together hides both. - A difference is not automatically an R bug. SAS orders missing numerics below all values, so if bmi < 18.5 classifies a missing BMI as underweight where R gives NA. Record those cases in equivalence_signoff.yaml as intentional_divergence. - The snapshot does not take haven out of the chain of custody, it confines it to one audited step. A misread is faithfully preserved in the parquet, so supply expect to validate the conversion against SAS-side PROC CONTENTS output rather than treating a clean round-trip as a correct read. No PHI in this repository Test fixtures are invented and say so. .gitignore blocks *.sas7bdat and *.parquet, with one deliberate exception for the synthetic inst/extdata/oracle_small.sas7bdat. The integration test against a real study runs only when HVTI_ORACLE_DIR points at real SAS datasets, a directory that lives outside this repository, and its assertions are on shape and verdicts only, so no patient value can reach a failure message.