---
title: "Getting Started with hvtiRutilities"
author: "John Ehrlinger"
date: today
format: html
vignette: >
  %\VignetteIndexEntry{Getting Started with hvtiRutilities}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5
)
```

## Introduction

The `hvtiRutilities` package provides utility functions for working with clinical research data at the Cleveland Clinic Heart, Vascular and Thoracic Institute (HVTI). It simplifies common data preparation tasks when importing and cleaning datasets, particularly those originating from SAS.

### Main Functions

- **`read_clinical_data()`**: Read SAS, CSV, Excel, or RDS files as they are; pass `convert_types = TRUE` to also apply `r_data_types()`
- **`r_data_types()`**: Automatically infer and convert data types based on content
- **`label_map()`**: Extract variable labels from labeled datasets into a lookup table
- **`get_label()`** / **`get_labels()`**: Safely look up one or many variable labels
- **`add_labels()`**: Register labels for derived variables on data frames or label maps
- **`apply_label_overrides()`**: Apply study-specific label overrides from a YAML file
- **`data_dictionary()`**: Build a type-annotated data dictionary from any labeled dataset
- **`proc_contents()`**: SAS `PROC CONTENTS` in R — variable metadata and completeness
- **`proc_means()`**: SAS `PROC MEANS` in R — numeric summaries, with SAS quantile definitions
- **`compare_datasets()`**: Summarise differences between two data pulls
- **`sample_data()`**: Generate labeled sample datasets for testing and examples

If you are coming from SAS, the [PROC CONTENTS and PROC MEANS in R](sas-procedures.html)
vignette walks the two ports side by side with the procedures they replace.

### Installation

```{r eval=FALSE}
# Install from GitHub
# install.packages("pak")
pak::pak("ehrlinger/hvtiRutilities")
```

```{r}
if (requireNamespace("hvtiRutilities", quietly = TRUE)) {
  library("hvtiRutilities") # Use installed package when available
} else {
  pkgload::load_all(export_all = FALSE, helpers = FALSE, quiet = TRUE)
}
```

## Basic Usage

### Automatic Type Conversion with `r_data_types()`

The `r_data_types()` function intelligently converts column types based on their content:

```{r}
# Generate sample data with various types
dta <- sample_data(n = 100)

# Examine original structure
str(dta)
```

Notice that the sample data has:
- `boolean`: integer values (1, 2)
- `logical`: character values ("F", "T")
- `char`: character values ("male", "female")

Now let's apply automatic type conversion:

```{r}
# Convert types automatically
dta_converted <- r_data_types(dta)

# Examine converted structure
str(dta_converted)
```

After conversion:
- `boolean`: converted to logical (TRUE/FALSE) because it has exactly 2 unique values
- `logical`: converted to factor (categorical variable)
- `char`: converted to factor (categorical variable)
- Continuous variables (`float`, `f_real`) remain numeric

### Transformation Rules

The function applies transformations in this order:

1. Character strings "NA", "na", "Na", "nA" → actual `NA` values
2. Numeric/integer with exactly 2 unique values → `logical`
3. Character columns → `factor`
4. Numeric with 3 to `factor_size` unique values → `factor`
5. Optionally: logical → `factor` (if `binary_factor = TRUE`)

## Working with Real Data

### Example: mtcars Dataset

Let's apply this to the built-in `mtcars` dataset:

```{r}
# Original mtcars
str(mtcars[, 1:5])
```

```{r}
# Apply type conversion
mtcars_clean <- r_data_types(mtcars)
str(mtcars_clean[, 1:5])
```

Notice:
- `cyl` (3 unique values: 4, 6, 8) → factor
- `vs` (2 unique values: 0, 1) → logical
- `am` (2 unique values: 0, 1) → logical
- `gear` (3 unique values) → factor
- `carb` (6 unique values) → factor

### Controlling Factor Conversion

Use `factor_size` to control when numeric variables become factors:

```{r}
# More strict: only convert if < 5 unique values
mtcars_strict <- r_data_types(mtcars, factor_size = 5)

# Check cylinder variable
class(mtcars_clean$cyl)  # factor (3 unique < 10)
class(mtcars_strict$cyl) # factor (3 unique < 5)

# Check carb variable
class(mtcars_clean$carb)  # factor (6 unique < 10)
class(mtcars_strict$carb) # integer (6 unique NOT < 5)
```

### Skipping Specific Variables

Sometimes you want to preserve certain variables in their original form:

```{r}
# Keep vs and am as numeric instead of converting to logical
mtcars_partial <- r_data_types(mtcars, skip_vars = c("vs", "am"))

# Compare
class(mtcars_clean$vs)    # logical (converted)
class(mtcars_partial$vs)  # numeric (preserved)
```

### Binary Variables as Factors

By default, binary variables become logical. Use `binary_factor = TRUE` to make them factors instead:

```{r}
mtcars_factor <- r_data_types(mtcars, binary_factor = TRUE)

# Compare
class(mtcars_clean$vs)   # logical
class(mtcars_factor$vs)  # factor
```

This can be useful for modeling or visualization where factor levels are preferred.

## Working with Variable Labels

Variable labels are common in clinical datasets, especially those imported
from SAS. `hvtiRutilities` provides a complete label management toolkit.
For a comprehensive guide including best practices and anti-patterns, see
`vignette("data-labels")`.

### Extracting and Looking Up Labels

```{r}
library(labelled)

# Generate labeled data (as would come from SAS import)
dta <- generate_survival_data(n = 100, seed = 42)

# Extract all labels into a lookup table
lmap <- label_map(dta)
head(lmap, 8)
```

Use `get_label()` for safe single-variable lookup --- it errors on typos
instead of silently returning `NA`:

```{r}
get_label(lmap, "age")
get_label(lmap, "lvefvs_b")
```

### Labeling Derived Variables

When you create new variables, label them immediately using `add_labels()`:

```{r}
dta$age_group <- cut(dta$age,
  breaks = c(0, 40, 60, Inf),
  labels = c("<40", "40-60", ">60")
)

# Label the data frame directly --- labels travel with the data
dta <- add_labels(dta, c(age_group = "Age Group at Surgery"))
var_label(dta$age_group)
```

### Labels Persist Through Transformations

Labels are preserved when using `r_data_types()`:

```{r}
dta <- sample_data(n = 50)
lmap_before <- label_map(dta)

dta_clean <- r_data_types(dta, skip_vars = "id")
lmap_after <- label_map(dta_clean)

identical(lmap_before, lmap_after)  # TRUE
```

## Complete Workflow Example

Here's a complete real-world workflow for preparing clinical data:

```{r}
# Step 1: Generate sample clinical data
set.seed(123)
clinical <- data.frame(
  id = 1:50,
  center = sample(c("Site A", "Site B", "Site C"), 50, replace = TRUE),
  treatment = sample(1:4, 50, replace = TRUE),
  age = round(rnorm(50, mean = 55, sd = 12)),
  sex = sample(c("M", "F"), 50, replace = TRUE),
  outcome = sample(0:1, 50, replace = TRUE, prob = c(0.3, 0.7)),
  followup_days = round(runif(50, 30, 730)),
  adverse_event = sample(c("None", "Mild", "Moderate", "Severe", "NA"),
                         50, replace = TRUE),
  stringsAsFactors = FALSE
)

# Step 2: Add variable labels (as from SAS import)
var_label(clinical$id) <- "Patient ID"
var_label(clinical$center) <- "Enrollment Center"
var_label(clinical$treatment) <- "Treatment Arm (1-4)"
var_label(clinical$age) <- "Age at Enrollment (years)"
var_label(clinical$sex) <- "Biological Sex"
var_label(clinical$outcome) <- "Primary Outcome (0=Failure, 1=Success)"
var_label(clinical$followup_days) <- "Days of Follow-up"
var_label(clinical$adverse_event) <- "Most Severe Adverse Event"

# Step 3: Clean and convert types
clinical_clean <- r_data_types(
  clinical,
  factor_size = 5,      # Only convert if < 5 unique values
  skip_vars = "id"      # Keep ID as integer
)

# Step 4: Examine results
str(clinical_clean)

# Step 5: Extract labels for reporting
label_lookup <- label_map(clinical_clean)

# Step 6: Use in analysis
# Count by treatment
table(clinical_clean$treatment)

# Outcome by treatment (using labels)
outcome_summary <- aggregate(
  outcome ~ treatment,
  data = clinical_clean,
  FUN = function(x) c(n = length(x), success = sum(x), rate = mean(x))
)
print(outcome_summary)
```

### Handling Missing Data

The function automatically handles character NA variants:

```{r}
# Data with various NA representations
messy <- data.frame(
  var1 = c("NA", "value1", "na", "value2", "nA"),
  var2 = c("NA", "A", "na", "B", "C"),
  var3 = c(1, 2, NA, 4, 5),
  stringsAsFactors = FALSE
)

clean <- r_data_types(messy)

# Character NAs converted to true NA
sum(is.na(clean$var1))  # 3 NAs
levels(clean$var1)       # Only "value1" and "value2"

sum(is.na(clean$var2))  # 2 NAs
levels(clean$var2)       # "A", "B", "C"
```

## Advanced Usage

### Custom Workflows for Specific Data Types

For datasets with specific requirements:

```{r}
# Lab results with reference ranges
labs <- data.frame(
  patient = 1:20,
  test_name = rep(c("Glucose", "HbA1c"), 10),
  value = c(rnorm(10, 100, 15), rnorm(10, 6.5, 1)),
  unit = rep(c("mg/dL", "%"), 10),
  flag = sample(c("Normal", "High", "Low"), 20, replace = TRUE),
  stringsAsFactors = FALSE
)

# Convert with specific settings
labs_clean <- r_data_types(
  labs,
  skip_vars = c("patient", "value"),  # Preserve ID and numeric values
  factor_size = 4                      # Conservative factor conversion
)

str(labs_clean)
```

### Integration with Data Import

`read_clinical_data()` reads a file as-is: types come back exactly as the
source stored them, and nothing is coerced by default. That default matters
for clinical data specifically — `r_data_types()` converts any two-valued
numeric column to `logical`, and a SAS 0/1 event or censoring flag is exactly
that shape. A survival model that expects `0`/`1` breaks silently if the flag
arrives as `TRUE`/`FALSE`. Opt into conversion explicitly with
`convert_types = TRUE`, and pass any `r_data_types()` argument (such as
`factor_size`) alongside it:

```{r eval=FALSE}
# Read SAS, CSV, or Excel — auto-detects format, no coercion
dta_raw <- read_clinical_data("path/to/data.sas7bdat")

# Opt in to type conversion, tuning it the same way you would call
# r_data_types() directly
dta <- read_clinical_data("path/to/data.sas7bdat",
                          convert_types = TRUE, factor_size = 15)

# Build a data dictionary for documentation
dict <- data_dictionary(dta)
write.csv(dict, "data_dictionary.csv", row.names = FALSE)
```

`read_clinical_data()` supports `.sas7bdat`, `.csv`, `.xlsx`, `.xls`, and
`.rds` files. `convert_types = FALSE` is the default; `factor_size` and the
other `r_data_types()` arguments are silently ignored unless
`convert_types = TRUE` is also passed.

## Best Practices

### When to Use `r_data_types()`

**Use it when:**
- Importing data from SAS, SPSS, or other statistical software
- Working with datasets where types aren't correctly inferred
- You have many categorical variables coded as integers
- You need consistent type handling across multiple datasets

**Skip it when:**
- Your data types are already correct
- You need very specific type conversions not covered by the function
- Working with specialized data structures (time series, spatial data, etc.)

### Recommended Settings by Use Case

**Exploratory Analysis:**
```{r eval=FALSE}
data_clean <- r_data_types(data, factor_size = 10)
```

**Modeling/Regression:**
```{r eval=FALSE}
data_clean <- r_data_types(data, factor_size = 5, binary_factor = FALSE)
```

**Descriptive Statistics/Tables:**
```{r eval=FALSE}
data_clean <- r_data_types(data, factor_size = 15, binary_factor = TRUE)
```

### Checking Results

Always verify the conversions make sense:

```{r eval=FALSE}
# Before
str(original_data)
summary(original_data)

# After
str(clean_data)
summary(clean_data)

# Check specific variables
table(clean_data$categorical_var)
```

## Summary

The `hvtiRutilities` package streamlines data preparation for clinical research:

- **`read_clinical_data()`**: Import from SAS, CSV, Excel, or RDS as-is, with type conversion available via `convert_types = TRUE`
- **`r_data_types()`**: Automatic, intelligent type conversion
- **`label_map()`**: Extract variable labels into a lookup table
- **`get_label()`** / **`get_labels()`**: Safe label lookup (single or vectorized)
- **`add_labels()`**: Register labels for derived variables
- **`apply_label_overrides()`**: Apply study-specific overrides from YAML (label maps or data)
- **`data_dictionary()`**: Build a type-annotated data dictionary
- **`proc_contents()`** / **`proc_means()`**: SAS `PROC CONTENTS` and `PROC MEANS` in R
- **`compare_datasets()`**: Audit differences between two data pulls
- **`sample_data()`** / **`generate_survival_data()`**: Generate labeled test data

Key features:
- Preserves variable labels through transformations
- Handles multiple NA representations
- Flexible control over factor conversion
- Works with data.frames, tibbles, and data.tables

For more detailed guides:
- Label handling: `vignette("data-labels")`
- Dataset versioning: `vignette("dataset-versioning")`
- Survival data: `vignette("survival-data")`
- Reproducible seeds: `vignette("reproducible-seeds")`
- GitHub: https://github.com/ehrlinger/hvtiRutilities
- Release notes: Run `hvtiRutilities.news()` in R

## Session Information

```{r}
sessionInfo()
```