Getting Started | R-(Shiny) meets Pavlovia Part One

Utilizing R and R-Shiny to improve Pavlovia-Workflow Part 1. Welcome to my new blog! Everything is a WIP ;)

Luke Bölling https://psychfactors.org (FernUniversität Hagen (Department of General Psychology))https://www.fernuni-hagen.de/psychologie-urteilen-entscheiden/team/luke.boelling.shtml
08-01-2021

Welcome to my new blog! I am Luke, research assistant at the FernUniversität Hagen (Department of General Psychology). As a remote university by design, Hagen was able to cope with the COVID-situation very well. The 14000 psychology students (18/19) and staff members were facing challenges, but remote procedures and systems could be adapted quickly and effectively. Switching to an effective remote/online experiment work-flow to get research progress without on-site-testing is one of these challenges.

The whole situation continues to be a learning process. Today and without further ado, I want to dive directly into some practical hacks for teachers and students to use the experiment hosting platform Pavlovia more effectively.

In a three (or more) part series I want to show you a few tricks to connect R and Pavlovia (or Unipark (Qualitrics) with Pavlovia) for an optimized online experiment workflow with multiple Gitlab-Users (e.g., students working on thesises and so on).

The whole series is most suited for experienced Pavlovia Users. After this rather advanced series, I am planning to release a more general introduction to online-experiments using Pavlovia.

Part 1:

In the first part, I’ll show you a handy way to download available data-files from Pavlovia directly into R (or in general Gitlab to R, without special packages)

1. Generate Access Token

Just generate a Personal Access token for your gitlab.pavlovia.org Account: https://gitlab.pavlovia.org/profile/personal_access_tokens Generate Access Token in Gitlab

Please be aware that this token is a fully fledged password-like credential for all your files on gitlab.pavlovia.org. Be careful with the generated token. Save the token to file named “token” in a folder where your R-Script is. IMPORTANT: Add the “token”-File to the gitignore File if you use GIT to manage your files. You do not want an access token to be uploaded to public repository.

2. Get the Project-Id of your repository

Now get the Project ID of your project in gitlab.pavlovia.org (here 104201). ProjectID

3. Create a R-File in the same directory as the “token” and load a few packages

4. Set up variables for the data-download-process

token <- read_file("token") # Personal Access Token for the Project
project_id <- 149 # Project ID

gitlabPavloviaURL <- paste0("https://gitlab.pavlovia.org/api/v4/projects/", 
                            project_id, 
                            "/repository/archive.zip") # API - URL to download whole repository

5. Use httr-Package to interact with the Gitlab-API of Pavlovia

r <- GET(gitlabPavloviaURL, add_headers("PRIVATE-TOKEN" = token)) # Getting Archive

bin <- content(r, "raw") # Writing Binary

temp <- tempfile() # Init Tempfile

writeBin(bin, temp) # Write Binary of Archive to Tempfile

listofFiles <- unzip(
  zipfile = temp, overwrite = T,
  junkpaths = T, list = T
) # Unzip only list of all files in the archive.zip file


csvFiles <- grep("*.csv", x = listofFiles$Name, value = T) 
# Grep only the csv Files (Pattern can be extended to get only data-csv file)

unzip(
  zipfile = temp, overwrite = T,
  junkpaths = T, files = csvFiles[1:100], exdir = "temp"
) # Unzip the first 100 csv Files in the temp-file in the temp-folder

csvFilesPaths <- list.files("temp/", full.names = T) 
# Get the unzipped csv-Files in the temp-directory

# To get only Valid CSV-Files and enable us to filter by 
# DateTime of the File: 
# We can parse the files standard date-time string in the Pavlovia-Default FileNames
dateTimeOfFiles <- tibble(filepaths = csvFilesPaths) %>%
  mutate(dateTime = str_extract(filepaths, "[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}h[0-9]{2}")) %>%
  filter(!is.na(dateTime)) %>%
  mutate(dateTime = parse_datetime(dateTime, "%Y-%m-%d_%Hh%M"))
# %>%  filter(dateTime > parse_datetime("2019-02-01_15h00", "%Y-%m-%d_%Hh%M"))
# This can be used to Filter by a specific time

dateTimeOfFiles
## # A tibble: 100 x 2
##    filepaths                                       dateTime           
##    <chr>                                           <dttm>             
##  1 temp/-99_stroop_2021-03-01_15h12.42.468.csv     2021-03-01 15:12:00
##  2 temp/ kj kj _stroop_2021-04-15_10h50.06.740.csv 2021-04-15 10:50:00
##  3 temp/00_stroop_2020-12-09_17h56.35.167.csv      2020-12-09 17:56:00
##  4 temp/00_stroop_2021-01-28_13h02.56.078.csv      2021-01-28 13:02:00
##  5 temp/00_stroop_2021-03-05_14h01.53.781.csv      2021-03-05 14:01:00
##  6 temp/00_stroop_2021-05-12_15h02.20.499.csv      2021-05-12 15:02:00
##  7 temp/00_stroop_2021-05-12_15h21.09.547.csv      2021-05-12 15:21:00
##  8 temp/000_stroop_2021-01-11_13h53.56.122.csv     2021-01-11 13:53:00
##  9 temp/000_stroop_2021-01-13_13h44.13.439.csv     2021-01-13 13:44:00
## 10 temp/000_stroop_2021-01-13_13h55.29.554.csv     2021-01-13 13:55:00
## # ... with 90 more rows

6. At this point we have all we need to get the data

# Now the read the desired data Files with purrr:
data <- tibble(filename = dateTimeOfFiles$filepaths) %>% 
  # create a data frame
  # holding the file names
  mutate(
    file_contents = map(
      filename, # read files into
      ~ read_csv(file.path(.))
    ) # a new data column
  )

# Unlink temp because we don't need it anymore
unlink("temp", recursive = T)
data
## # A tibble: 100 x 2
##    filename                                        file_contents
##    <chr>                                           <list>       
##  1 temp/-99_stroop_2021-03-01_15h12.42.468.csv     <spec_tbl_df~
##  2 temp/ kj kj _stroop_2021-04-15_10h50.06.740.csv <spec_tbl_df~
##  3 temp/00_stroop_2020-12-09_17h56.35.167.csv      <spec_tbl_df~
##  4 temp/00_stroop_2021-01-28_13h02.56.078.csv      <spec_tbl_df~
##  5 temp/00_stroop_2021-03-05_14h01.53.781.csv      <spec_tbl_df~
##  6 temp/00_stroop_2021-05-12_15h02.20.499.csv      <spec_tbl_df~
##  7 temp/00_stroop_2021-05-12_15h21.09.547.csv      <spec_tbl_df~
##  8 temp/000_stroop_2021-01-11_13h53.56.122.csv     <spec_tbl_df~
##  9 temp/000_stroop_2021-01-13_13h44.13.439.csv     <spec_tbl_df~
## 10 temp/000_stroop_2021-01-13_13h55.29.554.csv     <spec_tbl_df~
## # ... with 90 more rows

7. Get an overview of the data.

# Now we have anything the want:
# Get a overview of all available data-files
data %>%
  rowwise() %>%
  mutate(participant = list(file_contents$participant[1]), 
         fileDim = paste0("Rows:", dim(file_contents)[1], " Vars:", dim(file_contents)[2])[1])
## # A tibble: 100 x 4
## # Rowwise: 
##    filename            file_contents   participant fileDim  
##    <chr>               <list>          <list>      <chr>    
##  1 temp/-99_stroop_20~ <spec_tbl_df [~ <NULL>      Rows:0 V~
##  2 temp/ kj kj _stroo~ <spec_tbl_df [~ <chr [1]>   Rows:5 V~
##  3 temp/00_stroop_202~ <spec_tbl_df [~ <chr [1]>   Rows:12 ~
##  4 temp/00_stroop_202~ <spec_tbl_df [~ <chr [1]>   Rows:12 ~
##  5 temp/00_stroop_202~ <spec_tbl_df [~ <chr [1]>   Rows:1 V~
##  6 temp/00_stroop_202~ <spec_tbl_df [~ <chr [1]>   Rows:12 ~
##  7 temp/00_stroop_202~ <spec_tbl_df [~ <chr [1]>   Rows:12 ~
##  8 temp/000_stroop_20~ <spec_tbl_df [~ <chr [1]>   Rows:12 ~
##  9 temp/000_stroop_20~ <spec_tbl_df [~ <chr [1]>   Rows:7 V~
## 10 temp/000_stroop_20~ <spec_tbl_df [~ <chr [1]>   Rows:6 V~
## # ... with 90 more rows


Or merge everything in one file

# Read in all available data in a single tibble
data %>% select(file_contents) %>% # remove filenames, not needed anynmore
  unnest(cols = c(file_contents))
## # A tibble: 663 x 20
##    resp.keys resp.corr resp.rt trials.thisRepN
##    <chr>         <dbl>   <dbl>           <dbl>
##  1 right             1   0.151               0
##  2 right             0   1.35                0
##  3 right             0   0.364               0
##  4 right             0   1.14                0
##  5 right             1   0.939               0
##  6 right             0   2.23                0
##  7 left              1   0.640               0
##  8 right             1   2.14                0
##  9 right             1   0.851               0
## 10 down              1   0.884               0
## # ... with 653 more rows, and 16 more variables:
## #   trials.thisTrialN <dbl>, trials.thisN <dbl>,
## #   trials.thisIndex <dbl>, trials.ran <dbl>, text <chr>,
## #   letterColor <chr>, corrAns <chr>, congruent <dbl>,
## #   session <chr>, participant <chr>, date <chr>,
## #   expName <chr>, psychopyVersion <chr>, OS <chr>,
## #   frameRate <dbl>, fbclid <chr>

Get the gist of the file

You can get a Gist of the whole file here

Outlook

On the next part of the series, I will show you to use this Gitlab-Connection to build a simple Shiny-App to monitor the progress of your experiment.

Thank you for reading! Please feel free to contact me!

Luke

Next Part