In cleaning a dataset, I encode information about data missingness. One such category of missingness is “not asked”, for questions that were not displayed to participants due to their response to an earlier question.
If you use REDCap to gather data, the conditions used to control the display of questions are called “branching logic”. REDCap provides details of the branching logic in the data dictionary, which we can use to automate the encoding of this missingness category!
Step 0: Use the REDCap API
If you’re not using the REDCap API to access the data, let me introduce you to the REDCapR package. The key functions we will use are:
redcap_read() to access data
redcap_read_metadata() to access the data dictionary
This saves you having to repeatedly download the data and dictionary and avoids you storing data locally.
Let’s pretend we have the following dataset:
data <-tibble(record_id =c(1,2,3,4,5),age =c(16,18,20,24,26),has_pets =c(1,0,1,1,0),pet_rating =c(1,NA,2,1,NA),pet_insurance =c(NA,0,1,0,1)) |>mutate(across(!record_id, as.character))tt(data)
record_id
age
has_pets
pet_rating
pet_insurance
1
16
1
1
NA
2
18
0
NA
0
3
20
1
2
1
4
24
1
1
0
5
26
0
NA
1
Everything is a character at this point, which is relevant to be able to use dplyr::rows_update() later on.
We also have this dictionary from REDCap, noting this is only a few of the columns that REDCap provides:
With the data and dictionary read in to the environment, we can start step 1.
Step 1: Convert branching logic
Anatomy of REDCap branching logic
Let’s look at the branching logic for pet_rating, where has_pets must be 1 for the question to be shown.
[has_pets] = '1'
A variable (or field name as REDCap calls it) is always referred to using square brackets. This is true for every variable except for checkboxes, which also incorporate round brackets. I won’t go into checkboxes here — they’re annoying and require extra data cleaning.
Once you start combining conditions, REDCap uses brackets as you might expect. If the branching logic wanted to target people aged outside of 18 to 25 who have pets it would be:
([age] < '18' or [age] > '25') and [has_pets] = '1'
Parlez vous R, REDCap?
Now we understand the anatomy of REDCap branching logic, let’s compare how operators are expressed in REDCap vs. R:
Operator
REDCap
R
Greater than
>
>
Greater than or equals
>=
>=
Less than
<
<
Less than or equals
<=
<=
Equal
=
==
Not equal
<>
!=
And
and
&
Or
or
|
Note that ==, !=, & and | are not the same.
Nein, ich spreche R nicht
With our knowledge of how REDCap expresses variables and operators, we now need to translate into Rspeak.
Fortunately, this is made straightforward thanks to stringr::str_replace_all()
Note the pipe | needs to be escaped and that the spaces around and and or are intentional. REDCap requires spaces around these operators and by incorporating the spaces in the pattern we won’t accidentally target a variable that has and or or in its name
2
Instances where the branching logic looks for missing data are handled next, to prevent them being handled in parts 4 and 5. We’re targeting any combination of letters/numbers [:alnum:] and underscores contained within square brackets, that are followed by (not) equals to missing ''. By making use of regex groups \\1, if there are multiple conditions based on missingness within the same branching logic, they’ll all be handled.
3
Convert the remaining equals and not equals operators
4
This step targets how checkboxes are expressed in branching logic. It might depend on how you clean/name your variables, but I’ve included it as I ended up using it for my context.
5
Remove remaining square brackets.
The code annotations above explain each of the steps, but you can see the result below
field_name
field_type
branching_logic
converted_logic
record_id
text
NA
NA
age
text
NA
NA
has_pets
yesno
NA
NA
pet_rating
radio
[has_pets] = '1'
has_pets == '1'
pet_insurance
yesno
[has_pets] = '1' and [age] >= '18'
has_pets == '1' & age >= '18'
Step 2: Parse the converted logic as code
Before we use the logic, we need to do one more step. The branching logic as it currently stands will be TRUE if the criteria are met, which means the question was asked. We care about the question not being asked, and we also want to make sure we’re only going to affect observations with missing data.
To reflect this, we just need to add an is.na() condition and invert the converted logic expression using !()
is.na(pet_insurance) & !(has_pets == '1' & age >= '18')
Now we have the code in a format acceptable to R, we can use rlang::parse_expr() to execute the strings as code.
Since I’m iterating over many columns in my dataset, I wrote a function to return the rows that need to have their missing data encoded:
records_not_asked <-function(uncoded_data, column){ logic <- branching_logic |># Keep just the variable of interestfilter(field_name == column) |># Pull out the logicpull(not_asked) uncoded_data |># Apply the logic to the input data using parse_expr()filter(!!parse_expr(logic)) |># Update that column to be -888888mutate(field_name = column,value ="-888888") |># Return only record_id and input columnselect(record_id, field_name, value) }
The reason for returning data in this format is because I intend to use dplyr::rows_update() to update the original dataset, rather than try and do all of this in one pipe.
Step 3: Find records that need updating
With the function ready to iterate over columns that have branching logic, we can use purrr::map() to iterate over the data
With this list we can then safely update the values in the original dataset
data_updated <- data |>pivot_longer(cols =all_of(records_to_update$field_name),names_to ="field_name",values_to ="value" ) |>rows_update(records_to_update, by =c("record_id", "field_name")) |># Reshape the updated variables back to wide formatpivot_wider(names_from = field_name,values_from = value ) |># Order the variables as originally specifiedselect(all_of(names(data)))tt(data_updated)