---
title: "R Access to mDIS REST API"
author: "Knut Behrends"
date: "7/15/2019, updated Oct 2020"
output:
  html_document:
    keep_md: true
---

```{r setup, include=FALSE}
library(knitr)
knitr::opts_chunk$set(echo = TRUE)
knitr::opts_chunk$set(fig.path = "./figures/")

```

# mDIS REST API access with R

[Developer page](../../developer/)

#### Build your own reports and data analyses. Using the mDIS REST API, you can get mDIS data independently from any predefined forms and reports. All you need is a current login to mDIS.

This tutorial demonstrates how to get data from mDIS with R, and how to analyze some fake data from the ICDP JET Project. This dataset is not really downloadable at this time.



To load a similar  dataset, you need to run the `seed/example-dump` Yii [migration](/guide/developer/#yii-migration-scripts). This loads data from the ICDP DSEIS Project drilled in South Africa in 2017. You can also work with your own mDIS data, but then some URLs, URL parameters, axis labels and plot titles must be changed accordingly.

This tutorial does *not* demonstrate how to *edit* mDIS datasets (create, delete, duplicate, etc). Such API calls are [available](../), but this tutorial focuses on the simple case of getting data in a read-only mode.
If you need to learn to get write-access to the mDIS REST API, study the JS code in the [dis-data-gen repository](https://git.gfz-potsdam.de/icdp-osg/dis-datagen).

## What the R code does

- login to mDIS
- fetch some data
- convert it to a table (an R data frame)
- simple exploratory analysis

::: tip
Download the [RMarkdown File](./R-example.Rmd) directly, for usage in RStudio for example.
:::

### Necessary R packages

```{r pkgs, warning=FALSE, message=FALSE}
library(tidyverse)    # data processing + plotting helpers
library(jsonlite)     # read/write JSON
library(httr)         # give R web browsing capabilities
library(kableExtra)   # nicer tables
theme_set(theme_bw()) # ggplot layout theme
```

### Login to mDIS

Get credentials, and URL info, where to get the data from. Assign it to the `logindata` list.

We read it in as a JSON structure, because the mDIS REST API will return some more such JSON structures later. If you struggle with the following code block, later blocks will be difficult to deal with, too.

```{r creds}


my_c <- as_mapper( function(x, y) sprintf("%s=%s", x, y))
expedition <-c("Prees-2" = 4)
site <- c("Prees-2" = 5)
hole <- c("Prees-2" = 2)
qsp <- c(name = "core", "per-page" = 5000, "page" = 0,
         sort = "id",
         "filter[expedition_id]" = expedition, # JET
         "filter[site_id]" = site,       # Prees
         "filter[hole_id]" = hole)       # main hole
qsparams <- map2_chr(names(qsp), qsp, my_c) %>% str_c(collapse="&")

logindata <- jsonlite::fromJSON(txt = sprintf('{
  "username": "knb",
  "password": "knbpassword",
  "urlhost": "https://jet.rundis.com",
  "urlpaths": [
    "/api/v1/auth/login",
    "/api/v1/form?%s"
  ]
}', qsparams))

#logindata$password = Sys.getenv("MDIS_PW")
#jsonlite::toJSON(logindata)
```

You can insert your own password on line 10, or you can set it on line 18 (which is commented out at this time).

To actually connect and fetch data with the `logindata`, we need to create some helper functions first.

### Create Functions

Make an `mdis_api` function - a single function for login and data fetching.
This is suboptimal, but will suffice for now.

If the user is not logged in , the `logindata` list will *not* contain a `token` element. Thus we need to [log in](../auth-login.html) first, with data provided in `logindata`. We will receive a Security token ("Bearer Token"), which we will use as 32-character long "temporary password" from now on.

If the user is logged in, `logindata$token` will have a valid value, e.g. `9nxFYoXoBA7Eb6n1hc-FbwcTqeU5vf3a`. Then we can perform HTTP GET requests, sending the token as an extra HTTP header.

```{r mdis_api}
# function according to the httr vignette
# returns simple s3 object for easier debugging
# used for login and HTTP-GETting data
mdis_api <- function(logindata, path) {
  url <- modify_url(logindata$urlhost, path = path)
  if(is.null(logindata$token)){
    resp <- POST(url, ua = logindata$ua,
            body = list(username = logindata$username,
                        password = logindata$password))
  } else {
    resp <- GET(url,
                ua = logindata$ua,
                add_headers(
                  c(
                  Authorization = sprintf("Bearer %s", logindata$token))))
  }
  if (http_type(resp) != "application/json") {
    stop(str_c("API did not return json!",
               "Invalid mIDS filter setting (= wrong query string params)?    ",
               resp,
               sep ="\n"),
         call. = FALSE)
  }

  parsed <- jsonlite::fromJSON(content(resp, "text"), simplifyVector = FALSE)

  if (http_error(resp)) {
    stop(
      sprintf(
        "mDIS API request failed [%s]\n%s\n<%s>",
        status_code(resp),
        parsed[[1]],
        resp$url
      ),
      call. = FALSE
    )
  }

  # return simple S3 object, see comment below
  structure(
    list(
      content = parsed,
      path = path,
      response = resp
    ),
    class = "mdis_api"
  )
}

# Rather than simply returning the response as a list,
# I think it’s a good practice to make a simple S3 object.
# That way you can return the response and parsed object,
# and provide a nice print method.
# This will make debugging later on much much much more pleasant.

print.mdis_api <- function(x, ...) {
  cat("<mDIS ", x$path, ">\n", sep = "")
  str(x$content)
  invisible(x)
}


```

Actually log in. Get token and add it to `logindata` list.

```{r login}
# good practice: to identify yourself as user agent, ua
logindata$ua = user_agent("http://github.com/knbknb")

loggedin <- mdis_api(logindata, logindata$urlpaths[[1]])

logindata$token <- loggedin$content$token
```

### Fetch data

Make [API request](../form-index.html) to ``r logindata$urlpaths[[2]]`` to get all cores of a JET drillhole "Prees-2"  (site `r site["Prees-2"]`, hole `r hole["Prees-2"]`)

```{r tok}
parsed <- mdis_api(logindata, logindata$urlpaths[[2]])
```

JSON often contains JavaScript `null` values. In R, the `jsonlite` package converts these to R's built-in `NULL`. However in R, we prefer `NA` values instead. `NA` values are easier to work with than `NULL` objects.
Hence, replace NULLs with NAs in list parsed from JSON,
and put items in an R data frame, `cores_df`.

```{r req}
cores <- as.list(parsed)$content$items %>%
  map(function(x) map(x, function(y) ifelse(is.null(y), NA, y)))

cores_df <- map_df(cores, as_tibble, .name_repair = "minimal")

cores_df <- cores_df %>%
  mutate(core_ondeck = as.POSIXct(core_ondeck))
```

### The data

What does the table/data frame look like?

We got a `r sprintf("%sx%s", nrow(cores_df), ncol(cores_df))` table from the mDIS REST API.

Column names, data types, some example values:

```{r cores_df}
#skimr::skim(cores_df)
glimpse(cores_df)

```

A table view would be very, very wide so we list only the column definitions above, with some sample values.

## Analysis

Do some descriptive statistics with the drillcore data.

### Core Diameter Counts

We are studying drillhole `r hole["Prees-2"]`, site `r site["Prees-2"]` (ICDP JET project).
How "deep" did they drill?
(Remember, these are fake data)

```{r diam}
subtitle <- str_c("(fake data:) JET Core diameters (mm), Hole ", hole["Prees-2"], ", site ", site["Prees-2"], collapse = " ")
cores_df %>%
  group_by(core_diameter) %>%
  summarize(`max_depth (m)`= max(bottom_depth), n_Cores = n(), .groups = "keep")  %>%
  arrange(desc(core_diameter)) %>%
  ungroup() %>%
  knitr::kable(caption = subtitle, col.names = names(.)) %>%
  kableExtra::kable_styling(bootstrap_options = c("striped"))
```

### Who are the most active analysts/geologists?

```{r ana}
cores_df %>%
  count(analyst, sort =TRUE) %>%
  knitr::kable(caption = "Initials of active curators in the JET project", col.names = names(.)) %>%
  kableExtra::kable_styling(bootstrap_options = c("striped"))

```



### Core loss

Tabulating core loss

```{r coreloss}
table(as.Date(cores_df$core_ondeck), cores_df$core_loss_reason) %>%
  kable(caption = "When were JET cores lost? How many, and why?") %>%
  kableExtra::kable_styling(bootstrap_options = c("striped"))
```

The second column contains the number of cores recovered on this day.

### Core recovery in percent

See figure.

```{r lossfig}
cores_df %>% ggplot(aes(core_ondeck, core_recovery_pc/100, color=analyst)) +
  geom_jitter(alpha = 0.5) +
  geom_hline(yintercept = 1, color = "brown", linetype = 2, alpha = 0.5) +
  labs(title = "Fake Data (!): Prees-2 Core Recovery in Percent",
       subtitle = subtitle,
       y = "% Core Recovered",
       x = "Day of Recovery (2020)") +
  scale_y_continuous(labels = scales::percent_format(accuracy = NULL),  breaks = seq(0, 1.2, 0.1))
```

Values larger than 100% are possibly due to solid core material broken off a bit below the core barrel. Sometimes cores are retrieved with a bit of extra rock sticking out of the pipe.
Alternatively it might be elastic material (mud) decompressing and expanding in a plastic liner.

Many more analyses are now possible. These were just illustrations.

## REST API

### Documentation

See [REST API](../) page for extensive documentation.

[Developer page](../../developer/)
