mDIS User Documentation

REST API

Developer page

New in 2023

OpenAPI-style documentation of the mDIS REST API (opens new window).

mDIS has two REST (opens new window) API endpoint patterns

  1. a "regular" REST API endpoint pattern: URLs starting with /api/v1/<controllername>?, such as /api/v1/form?, for querying and manipulation of science-data, the focus of this article
  2. a code-generation API endpoint pattern: URLs starting with /cg/api/v1/, e.g. /cg/api/summary?. These endpoints are for "privileged" data definition operations such as Create Table/Models, Create Form, Generate PHP-code, etc.

Almost all endpoints take query string parameters (or POST data) as arguments. Query string parameters are used for filtering data. JSON data in POST/PUT/DELETE requests contains the payload for creating, updating and deleting records.
For a listing of the acceptable arguments, click on the respective links in the Controller Documentation below.

As a reminder, here is a simplified view of the mDIS client-server architecture:

client-server-diagram

Implementation Notes

Data Format

The API returns JSON data.

Versioning

The science-data mDIS REST API supports versioning. The current version is v1.

Security

Authentication is handled via usernames and passwords.

After successful authentication an OAuth2 Bearer Token (opens new window) is granted. It is valid for 1 day, but the duration of validity can be refreshed and extended.

API calls to REST API endpoints must submit that Bearer token in the HTTP header of each request: Authorization: Bearer abEARERtOKENlOOKSlIKEtHIS.

Directories

The REST API implementations reside in directories backend/modules/api/modules/v1/controllers and backend/modules/cg/controllers. Both call classes from the yii\rest namespace and from Gii (opens new window), the automatic code generator that comes with the Yii framework. Gii is an optional extension (opens new window) of the Yii framework. Gii's role in mDIS is explained here. The api and cg Modules and the Gii extension are being cleverly loaded ("bootstrapped" (opens new window)) by Yii. See file backend/config/web.php.

Happy Paths only

In the following API examples, only successful REST-API calls are documented - the "Happy Path Scenario".

Error Messages

Any HTTP Server Error responses will not be documented in great detail here. Error messages result from edge cases, and their structure is always very similar and thus highly redundant.
In the mDIS GUI, however, error messages are shown as a notification.
In mDIS forms, this notification appears as a yellow popup box in the upper right corner of the screen. Proper, shortened textual descriptions are shown, as well as low-level Error Messages. In API calls, notifications and error information can be seen in the browser's Developer Console.

Controllers

Internally to mDIS, REST-API calls are mapped to controller method calls. Available mDIS controllers are explained below.

The term "Controller" mentioned below and in the sidebar refers to the Model–View–Controller (opens new window) idiom of software architecture. A controller class accepts input and converts it to commands for the models (here, database tables) or views (here, Webpages or JSON-data-chunks).

URL Routing

To understand how the REST API works, it is helpful but not required to learn which URL paths the controllers understand, that is, how URL routing mechanisms work inside Yii2. A quick look into file backend/config/web.php, section 'urlManager' => [...] helps a lot. Also look into backend/modules/api/Module.php and backend/modules/cg/Module.php which contain many important URLManager rules.

Postman collection

There exists an ICDP-internal "testsuite" for REST API calls, a demonstrating selected mDIS REST-API calls mentioned below. The collection is available as a Postman (opens new window) collection. It can automate create, inserts, updates, deletes, gets, logins. It also contains many tests and a test runner. This collection is available to all ICDP team members as a Postman Shared Workspace.

Open Endpoints

Open endpoints do not require Authentication.

Login

  • Login POST /api/v1/auth/login

Reminder

At this time, the Login Screen is the only endpoint that does not require any authentication.

However its sole purpose is to accept login requests with username and password, and to give out security tokens, enabling the user to proceed to closed endpoints. After successful login, the web server returns a Bearer token in the JSON response:

{
  "username": "sly",
  "token": "lVAdQWaTrw2dO-aXwcxTqzIR0Pedx_Cb",
  "roles": [ "developer" ]
}
1
2
3
4
5

(Fragment, simplified)

Endpoints with authentication

"Closed" endpoints require as valid "Bearer Token" to be included in the header of the request. A Token can be acquired from the Login response above. The token acts like a temporary password, or as a temporary API key. A benefit of the Bearer token is that the "real" username and password are submitted only once, during login. Later, instead, the token is submitted with each HTTP request.

All endpoints that require authentication return an error response of Unauthorized, HTTP status 401 (opens new window):

{
  "name": "Unauthorized",
  "message": "Your request was made with invalid credentials.",
  "code": 0,
  "status": 401,
  "type": "yii\\web\\UnauthorizedHttpException"
}
1
2
3
4
5
6
7

However, HTTP status 422 ("Unprocessable Entity") (opens new window) is also returned often, e.g. when invalid filter expressions are used as query string parameters.

A successful authentication response looks like this.

{
  "id": 3,
  "username": "bnk",
  "email": "bnk@example.com",
  "token": "f5uKee5XVMB6X666eaaugFr_Aw-YCMz4", //temporary password
  "roles": ["_administrator_"],
  "permissions": [
    "form-archive-file:edit",
    "form-archive-file:view",
    "form-cores:edit",
    "form-cores:view",
    "form-init-gas:edit",
    "form-init-gas:view",
    "form-section:edit",
    "form-section:view"
  ]
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Note that an authentication token was provided, which is valid until it expires, or until the user logs out. Expiration date can be extended simply by reloading any mDIS webpage.

Auth Controller

  • Login POST /api/v1/auth/login - see above
  • Logout POST /api/v1/auth/logout - returns 1 if successful (= Security Token was still valid)
  • Refresh GET /api/v1/auth/refresh

App Controller

This lists the forms created by mDIS admins. The result of this call is used to construct the listing in the sidebar of the mDIS dashboard.

  • Forms GET /api/v1/app/forms

There are no other API endpoints to change the forms-listing. However, another useful URI to know is /cg/api/summary, which returns a complex object with keys modules, models, forms which returns read-only metadata (such as the column names) of tables, tablesets (internally named modules), models, and forms.

Global Controller (CRUD controller)

The endpoints under global are used to perform CRUD operations (Create, Retrieve, Update, Delete, Defaults, ..etc) for data models under directory /backend/models. Mainly to manage ArchiveFiles and ListValues data models, and also to get data from all columns of the datatables. This is in contrast to the forms controller which only returns the subsets that end users users can actually see with that form.

Form Controller

Form is a specialization of global. The endpoints under form are used to manage (create, update, delete, defaults, ...etc) data models under directory /backend/forms which inherit from the data models in directory /backend/models. They usually only return a subset of the data.

Example. A form controller URL looks like this:

/api/v1/form?per-page=1000&page=1&sort=core&filter[hole_id]=${hole_id}&&name=core

Substitute the $hole_id with the value of the Primary Key id column for that hole; e.g. if the id for hole 2 of a certain site has the value 15, use filter[hole_id]=15.

List Values Controller

List Values is also a specialization of global i.e. it contains all the actions in global (except filter-list & reports) and the following extra ones:

  • List Names GET /api/v1/list-values/list-names - Names of Select-Lists
  • List Values GET /api/v1/list-values - Values for each Select-List

File Controller

Used to manage uploaded files, and to connect or attach them to existing datasets.

  • Index GET /api/v1/file
  • Assign POST /api/v1/file/assign
  • Delete POST /api/v1/file/delete
  • Upload POST /api/v1/file/upload
  • MetaData GET /api/v1/file/meta-data/<name>
  • Unassign PUT /api/v1/file/unassign/<fileId>

Code Examples for using these REST API calls with curl can be found in some mDIS-installer scripts (opens new window) for customizing the mDIS dashboard.

TBC Widgets Controller

Dashboard widgets controller, manages visibility, alignment, cloneability.

The code examples mentioned in the previous paragraph demonstrate some calls.

TBC

TBC Importer Controller

CSV file controller, ValueList controller.

TBC

TBC Posts Controller

TBC - Dashboard Posts controller

Missing REST API Endpoints

These items do not have a REST API endpoint, or documentation is missing:

  • Reports. There is no ReportsController. Reports can be listed via the Form Controller or the Global Controller though. However new reports cannot be created via a REST API.
  • Everything covered by the cg-API (see below) has no Documentation at this time.
  • User management. Creating users and roles are basically one-off tasks and not very data-intensive. There is no REST API for managing users. (It might still be scriptable with PHP, though)

Physical Implementation

Filesystem Locations

The most important PHP file governing API functionality is backend/modules/api/Module.php. The bootstrap() Method in this file decides which REST endpoints are available, and which standard HTTP methods (GET/PUT/DELETE,...) are supported.

The PHP bases classes actually implementing the endpoints and methods supported by the API are stored in backend/modules/api/common/controllers, whereas the classes in backend/modules/api/modules/v1/controllers are simple wrapper-classes that only extend the base classes (without overriding anything).

PHP Third-Party Libraries

Authentication and all security matters are handled by the Yii framework.

See vendor/yiisoft/yii2/filters/auth/HttpBearerAuth and vendor/yiisoft/yii2/filters/VerbFilter for details. The Sysadmin page contains a big-picture list of third-party libraries. (Look for "dektrium").

TBC

Tutorials

The following tutorials show how the mDIS can be accessed without a Webbrowser, without using the preconfigured data input forms. Following along these tutorials is interesting for developers.

All tutorials perform a simple task: logging in and fetching data. No tutorial performs a more complex action (e.g. smart-copying an existing dataset) at this time.

bash

bash script: for quick and dirty command-line processing on Linux, with curl and jq. Also see example below on this page.

Recommended. Do this tutorial, because nearly all other tutorials need some shell scripting, sooner or later.
Moreover, almost all programmers prefer to save often-needed commands, with preferred command line options, in small shellscripts.

Powershell

Powershell 6 script: for quick and dirty command-line processing on Windows. Here, csv file export.

R

R tutorial: data access and some simple analysis.

Javascript

There is a tutorial using ES6 features such as Promises. It's actually a set of tutorials. Needs rework

More languages coming soon

More tutorials for Python, Perl, Go, PHP...

cg API

cg = Code-Generation

The cg-API is designed for advanced tasks and internal use, during interactive work with the Template manager. Most endpoints (except /cg/api/summary) exposed by cg require privileged access with the developer role or the sa role. Therefore they are not accessible to external users, and they are more complicated to use.

The internals of the cg-API are not explained here. Still, check out the following paragraphs.

TBC

Pro Tip

Some of capabilities of the cg-API are disabled by design. They become only available when certain environment variables are set, and even then this functionality is only available on the command line.

One example is a "drop everything and rebuild everything from scratch" set of actions which is only useful at installation time.

The environment variable is YII_ENV=dev, and the actions can be listed via the yii help command.

Useful URLs / endpoints

  • /cg/api/summary, which returns a complex object with keys modules, models, forms which returns read-only metadata about tablesets (internally named modules), models, and forms. This method is very useful if need get a table definition (even from an empty table) in JSON form, without retrieving any records. This endpoint can be called by any user.
  • /cg/api/verify-uploaded-template - imports a table-definition .json file. Prerequisite to create a new model. Developer and administrator roles only. - The light-blue sidebar of the Template-Manager ("Upload *.json file") is a graphical representation of this API.
  • /cg/api/delete-model?name=... - deletes an empty table from the database, and removes a corresponding .json file from the file system (specifically, from backend/dis_templates/models/). Developer and administrator roles only.

How to get a table/model definition in JSON form with bash

Only relevant commands are shown. For the entire shellscript, see the Bash tutorial.

#/bin/bash
# ...
url2="$host/cg/api/summary"
json2=$(curl -s -H "Content-Type:application/json" \
-H "Authorization: Bearer $token" $url2)

## process JSON data with jq
echo "$json2" | jq '.models[]| select(.table == "core_section") | .columns'
1
2
3
4
5
6
7
8

Result: (column names of table core_section)

[
  "id",
  "core_id",
  "section",
  "combined_id",
  "top_depth",
  "section_length",
  "bottom_depth",
  "analyst",
  "igsn",
  "section_state",
  "curated_length",
  "comment",
  "ukbgs_section_id",
  "ukbgs_section_extra"
]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

For real-world use of the cg-API, some knowledge about Gii (opens new window), the Yii2 extension for code generation, is recommended.

TBC

mDIS Developer pages