--- title: "Start a new study from a delivered dataset" author: "John Ehrlinger" vignette: > %\VignetteIndexEntry{Start a new study} %\VignetteEngine{quarto::html} %\VignetteEncoding{UTF-8} format: html: toc: true embed-resources: true --- This tutorial starts with an empty directory and an analysis dataset that has just been delivered. It takes the study through setup, data registration and a first descriptive job, then through one analysis chain, so you can see how the templates hand work to one another. If the study directory already exists and holds jobs and data, start with [Adopt an existing study](study-setup.html) instead. That guide keeps the existing layout and files. The executable example uses a 40-row synthetic dataset generated below. Its files stay in a temporary directory and disappear after the article renders. ## Before you start A delivered dataset here means one analysis-ready file, CSV or SAS transport, already built from the source extract. Building that file is a separate step; this guide begins once it exists. Keep it on the study share: patient data never enter a git repository. The study itself must exist in Study Tracker before its directory does. Create the study record there first and note its Study Tracker ID: every later step takes its identity from that record. ```{r} #| label: synthetic-setup #| include: false knitr::opts_chunk$set(collapse = TRUE, comment = "#>") library(hvtiRutilities) library(hvtiRtemplates) workspace <- withr::local_tempdir(.local_envir = environment()) new_root <- file.path(workspace, "new-study") i <- seq_len(40L) delivered <- data.frame( ccfid = i, dead = i %% 2L, iv_dead = i / 10, age = 40 + i, female = i %% 2L ) ``` ## Create the study Create the directory with the `study-setup` command from `qhsprograms`, giving it the Study Tracker ID and the directory name. Preview first; the dry run writes nothing: ```sh study-setup --dry-run 42 example_study study-setup 42 example_study ``` `study-setup` reads the Tracker record, read-only, and turns the new directory into a study. It writes `_study.yml`, which carries the study's name and Tracker ID, a README linking the Tracker record, an RStudio project file, environment defaults, `renv`, and the numbered folders every job is placed in: | folder | holds | |---|---| | `00_datasets` | the registered data | | `10_descriptive` | descriptive tables and checks | | `20_distributions` | life tables and hazard models | | `30_analyses` | multivariable and machine-learning models | | `40_graphs` | figures | | `50_documents` | Word tables and manuscript material | | `90_estimates` | saved output that other jobs read | `90_estimates` is numbered last on purpose. It holds results, not jobs, and a job that needs another job's result reads it from there. Now open the directory in RStudio as a Project (**File > New Project > Existing Directory**) and work from its `.Rproj` from then on. Select the R version and load the hvtiR packages as described in [Adopt an existing study](study-setup.html#open-the-study-as-an-rstudio-project): ```r library(hvtiRutilities) library(hvtiRtemplates) ``` Underneath, `study-setup` calls `hvtiRutilities::study_setup()` with the Tracker record's values. This article cannot reach Study Tracker while it renders, so the executable example calls `study_setup()` directly, in a temporary directory: ```{r} #| label: create-study # A vignette builds a throwaway study; in a real study, run hvtiR::setup(). study_setup(new_root, study = "Synthetic new study", study_tracker_id = 42L, renv = FALSE) sort(list.files(new_root)) ``` ## Register the dataset Copy the delivered file into `00_datasets`, then register it. Registration records the file's size, checksum and the population it describes in `manifest.yaml`: ```r register_data( built = "built.csv", role = "study", population = "Patients meeting the study inclusion criteria" ) ``` Every job reads data through this registration rather than through a file path. That is what lets a rendered job record exactly which bytes it read: a job that opens a path directly cannot tell you, a year later, which version of the file it saw. ```{r} #| label: register-data utils::write.csv( delivered, file.path(study_dir("datasets", new_root), "built.csv"), row.names = FALSE ) register_data( new_root, built = "built.csv", role = "study", population = "Synthetic full cohort" ) ``` Registration names no endpoint and no cohort. Those belong to the jobs that analyze them, so one registered dataset can serve every job in the study. ## Scaffold the first EDA job Start with the general descriptive checks: ```r general <- open_job("dc", subject = "cohort", type = "eda", qualifier = "general") ``` `open_job()` creates the job and opens it in RStudio. The file is named `....qmd`, here `10_descriptive/dc.general.cohort.eda.qmd`. The subject names what the job is about and need not be a statistical endpoint: `cohort` describes the whole registered population. The type names the stage, and the pair keeps a set of related jobs and their outputs together. The executable example scaffolds without opening an editor: ```{r} #| label: scaffold-eda general <- add_job("dc", subject = "cohort", type = "eda", dir = new_root, qualifier = "general") sub(paste0(new_root, "/"), "", general, fixed = TRUE) ``` Each job carries `EDIT:` markers where the template cannot know the study's answer: variables, labels, groups. Work through them from the top. A job that still contains one is unfinished. The first `add_job()` in a study also installs its provenance hooks. It adds pre-render and post-render entries to `_quarto.yml`, keeping any settings already there, and copies the hook scripts into `.hvtiR/hooks/`: ```{r} #| label: hooks cat(readLines(file.path(new_root, "_quarto.yml")), sep = "\n") list.files(file.path(new_root, ".hvtiR"), recursive = TRUE) ``` ## Render and read the provenance Render the job with the **Render** button, or from the Console: ```r render_job(general, final = TRUE) ``` Both run the same hooks. While the job executes it records what it read; once Quarto has written the HTML, the post-render hook publishes a sidecar beside it, `dc.general.cohort.eda.provenance.json`. An abridged example from a synthetic study: ```json { "job": "dc.general.cohort.eda", "rendered": "2026-09-23T14:42:04Z", "study": { "name": "Synthetic new study", "file": "_study.yml", "sha256": "35f29fb1..." }, "r": { "version": "4.6.1", "platform": "aarch64-apple-darwin23" }, "data": [ { "dataset": "study", "path": "00_datasets/built.csv", "role": "study", "bytes": 1051, "sha256": "3d6f6b0c..." } ], "artifacts": [], "source": "10_descriptive/dc.general.cohort.eda.qmd", "subject": "cohort", "type": "eda", "output": { "file": "dc.general.cohort.eda.html", "sha256": "95455dd3..." } } ``` Read it as a receipt. `data` names the registered file and the checksum of the bytes the job read. Its `dataset` says `"study"` although the job says `DATASET <- "built"`: the two names mean the same dataset, and records always use `"study"`, so a sidecar written by an older job compares equal. `packages` and `renv_lock`, omitted here, record every package version and the study's lockfile. `output` carries the checksum of the HTML it describes, so a sidecar can be matched to its report and to nothing else. If a render fails, the report and sidecar already on disk are left as they were. A new sidecar is only ever published beside the HTML it describes. One job renders at a time in a study; a second render started while one is running stops and asks you to wait. ## A first analysis chain: death Most analyses are chains: one job saves a result that the next reads. The hazard chain has three jobs. `ac` computes the actuarial life table, `hz` fits the parametric hazard model, and `hp` plots the two together. ```r open_job("ac", subject = "death", type = "hz") open_job("hz", subject = "death", type = "hz") open_job("hp", subject = "death", type = "hz") ``` All three share one subject and one type, and that is what connects them. The pair `("death", "hz")` names a set, and each job writes and reads its saved results in the set's own folder, `90_estimates/death-hz/`. `hp` finds `ac.rds` and `hz.rds` there without a path to edit. A second analysis of the same endpoint, a random survival forest say, takes a different type and so a different folder; the two chains cannot overwrite each other's life table. ```{r} #| label: scaffold-chain chain <- c( ac = add_job("ac", subject = "death", type = "hz", dir = new_root), hz = add_job("hz", subject = "death", type = "hz", dir = new_root), hp = add_job("hp", subject = "death", type = "hz", dir = new_root) ) sub(paste0(new_root, "/"), "", chain, fixed = TRUE) ``` The filenames read `ac.death.hz.qmd`, `hz.death.hz.qmd` and `hp.death.hz.qmd`: the template, then subject and type, so a study's jobs sort by template. `hz` appears twice in the second because the set is named for its method and the job is that method's template. ```{=html}
The death hazard chain: ac and hz save handoffs that hp reads ac.death.hz.qmd actuarial life table hz.death.hz.qmd parametric hazard ac.rds 90_estimates/death-hz/ hz.rds 90_estimates/death-hz/ hp.death.hz.qmd overlay plot each job also publishes a .provenance.json beside its HTML
Solid boxes are jobs; dashed boxes are the saved handoffs they share through the set's folder.
``` Render in chain order: `ac`, then `hz`, then `hp`. Each saved handoff carries its own lineage, the registered data it was built from, so `hp` can check its inputs before it plots them. It stops, with a message saying which job to rerun, when a handoff has no lineage or when `ac` and `hz` were built from different data. Rerender both upstream jobs after the registered data change. `hp`'s sidecar then lists the data both handoffs were built from and each handoff's checksum. ## Where to go next `template_list()` lists every template with its prefix, qualifier and folder: ```{r} #| label: templates head(template_list()[, c("name", "prefix", "qualifier", "folder")], 10) ``` The other chains follow the same pattern: a fitting job saves a handoff that later jobs read. The random forest jobs (`rfc`, `rfr`, `rfs`) pair a `-fit` job with an `-explain` job; the logistic family (`lm`) fits and validates outcome and propensity models; `bh` reports bootstrap variable selection.