REST API

Developer page

API Reference

The api-reference folder contains detailed) documentation for various API endpoints available in the mDIS app.

Conceptual Overview

Below is a more conceptual overview of the mDIS REST API, including links to more information about the API, its implementation, and some tutorials.

New in 2023

OpenAPI-style documentation of the mDIS REST API.

(You need to be logged in to the ICDP GitLab instance to see that repo.)

mDIS has two REST API endpoint patterns

1. a "regular" REST API endpoint pattern: URLs starting with /api/v1/<controllername>?, such as /api/v1/form?, for querying and manipulation of science data, the focus of this article
2. a code-generation API endpoint pattern: URLs starting with /cg/api/v1/, e.g. /cg/api/summary?. These /cg/... endpoints are for "privileged" data definition operations such as Create Table/Models, Create Form, Generate PHP code, etc.

Almost all endpoints take query string parameters (or POST data) as arguments. Query string parameters are used for filtering data. JSON data in POST/PUT/DELETE requests contains the payload for creating, updating, and deleting records.
For a listing of the acceptable arguments, click on the respective links in the Controller Documentation below.

As a reminder, here is a simplified view of the mDIS client-server architecture:

Implementation Notes

Data Format

The API returns JSON data.

Versioning

The science data mDIS REST API supports versioning. The current version is v1.

Security

Authentication is handled via usernames and passwords.

After successful authentication, an OAuth2 Bearer Token is granted. It is valid for 1 day, but the duration of validity can be refreshed and extended.

API calls to REST API endpoints must submit that Bearer token in the HTTP header of each request: Authorization: Bearer abEARERtOKENlOOKSlIKEtHIS.

Directories

The REST API implementations reside in directories backend/modules/api/modules/v1/controllers and backend/modules/cg/controllers. Both call classes from the yii\rest namespace and from Gii, the automatic code generator that comes with the Yii framework. Gii is an optional extension of the Yii framework. Gii's role in mDIS is explained here. The api and cg Modules and the Gii extension are being cleverly loaded ("bootstrapped") by Yii. See file backend/config/web.php.

Happy Paths only

In the following API examples, only successful REST API calls are documented - the "Happy Path Scenario".

Error Messages

Any HTTP Server Error responses will not be documented in great detail here. Error messages result from edge cases, and their structure is always very similar and thus highly redundant.
In the mDIS GUI, however, error messages are shown as a notification.
In mDIS forms, this notification appears as a yellow popup box in the upper right corner of the screen. Proper, shortened textual descriptions are shown, as well as low-level error messages. In API calls, notifications and error information can be seen in the browser's Developer Console.

Controllers

Internally to mDIS, REST API calls are mapped to controller method calls. Available mDIS controllers are explained below.

The term "Controller" mentioned below and in the sidebar refers to the Model–View–Controller idiom of software architecture. A controller class accepts input and converts it to commands for the models (here, database tables) or views (here, web pages or JSON data chunks).

URL Routing

To understand how the REST API works, it is helpful but not required to learn which URL paths the controllers understand, that is, how URL routing mechanisms work inside Yii2. A quick look into file backend/config/web.php, section 'urlManager' => [...] helps a lot. Also look into backend/modules/api/Module.php and backend/modules/cg/Module.php, which contain many important URLManager rules.

Postman collection

There exists an ICDP-internal "testsuite" for REST API calls, demonstrating selected mDIS REST API calls mentioned below. The collection is available as a Postman collection. It can automate create, inserts, updates, deletes, gets, logins. It also contains many tests and a test runner. This collection is available to all ICDP team members as a Postman Shared Workspace.

Physical Implementation

Filesystem Locations

The most important PHP file governing API functionality is backend/modules/api/Module.php. The bootstrap() method in this file decides which REST endpoints are available, and which standard HTTP methods (GET/PUT/DELETE,...) are supported.

The PHP based classes actually implementing the endpoints and methods supported by the API are stored in backend/modules/api/common/controllers, whereas the classes in backend/modules/api/modules/v1/controllers are simple wrapper classes that only extend the base classes (without overriding anything).

PHP Third-Party Libraries

Authentication and all security matters are handled by the Yii framework.

See vendor/yiisoft/yii2/filters/auth/HttpBearerAuth and vendor/yiisoft/yii2/filters/VerbFilter for details. The Sysadmin page contains a big-picture list of third-party libraries. (Look for "dektrium").

TBC

Tutorials

The following tutorials show how the mDIS can be accessed without a web browser, without using the preconfigured data input forms. Following along these tutorials is interesting for developers.

All tutorials perform a simple task: logging in and fetching data. No tutorial performs a more complex action (e.g. smart-copying an existing dataset) at this time.

bash

bash script: for quick and dirty command-line processing on Linux, with curl and jq. Also see the example below on this page.

Recommended. Do this tutorial, because nearly all other tutorials need some shell scripting, sooner or later.
Moreover, almost all programmers prefer to save often-needed commands, with preferred command line options, in small shell scripts.

Powershell

Powershell 6 script: for quick and dirty command-line processing on Windows. Here, CSV file export.

R

R tutorial: data access and some simple analysis.

Javascript

There is a tutorial using ES6 features such as Promises. It's actually a set of tutorials. Needs rework

More languages coming soon

More tutorials for Python, Perl, Go, PHP...

cg API

cg = Code-Generation

The cg-API is designed for advanced tasks and internal use, during interactive work with the Template manager. Most endpoints (except /cg/api/summary) exposed by cg require privileged access with the developer role or the sa role. Therefore, they are not accessible to external users, and they are more complicated to use.

The internals of the cg-API are not explained here. Still, check out the following paragraphs.

TBC

Pro Tip

Some of the capabilities of the cg-API are disabled by design. They become only available when certain environment variables are set, and even then this functionality is only available on the command line.

One example is a "drop everything and rebuild everything from scratch" set of actions which is only useful at installation time.

The environment variable is YII_ENV=dev, and the actions can be listed via the yii help command.

Useful URLs / endpoints

• /cg/api/summary, which returns a complex object with keys modules, models, forms which returns read-only metadata about tablesets (internally named modules), models, and forms. This method is very useful if you need to get a table definition (even from an empty table) in JSON form, without retrieving any records. This endpoint can be called by any user.
• /cg/api/verify-uploaded-template - imports a table-definition .json file. Prerequisite to create a new model. Developer and administrator roles only. - The light-blue sidebar of the Template Manager ("Upload *.json file") is a graphical representation of this API.
• /cg/api/delete-model?name=... - deletes an empty table from the database, and removes a corresponding .json file from the file system (specifically, from backend/dis_templates/models/). Developer and administrator roles only.

How to get a table/model definition in JSON form with bash

Only relevant commands are shown. For the entire shell script, see the Bash tutorial.

#/bin/bash
# ...
url2="$host/cg/api/summary"
json2=$(curl -s -H "Content-Type: application/json" \
-H "Authorization: Bearer $token" $url2)

## Process JSON data with jq
echo "$json2" | jq '.models[] | select(.table == "core_section") | .columns'

Result: (column names of table core_section)

[
  "id",
  "core_id",
  "section",
  "combined_id",
  "top_depth",
  "section_length",
  "bottom_depth",
  "analyst",
  "igsn",
  "section_state",
  "curated_length",
  "comment",
  "ukbgs_section_id",
  "ukbgs_section_extra"
]

For real-world use of the cg-API, some knowledge about Gii is recommended.

TBC

mDIS Developer pages