mDIS For Developers

Roles and Permissions

There are two aspects of working as an mDIS developer:

Software Developer: Managing an mDIS system, making changes to the mDIS source code, as a software developer with write-access to mDIS sourcecode files and to the mDIS database backend. The user can change everything inside mDIS.
Predefined Role 'developer': Interacting with mDIS as a user having the predefined role developer according to the Role-Based-Access-Control (RBAC) User Management feature. The 'developer' role is actually less powerful than the 'administrator' role. ('administrator' can manage permissions, 'developer' cannot). See mDIS User Management for details.

This section describes both aspects, but focuses mainly on the Software Developer role.

Note

Many tasks and responsibilities overlap with the reponsibilities of an "mDIS sysadmin". A clear separation is not always possible.

Operating System: Users, Permissions

In the server backend, the equivalent to the mDIS Frontend Role developer is the operating system account administrator or mdis. Substitute your preferred user-account name here. Depending on your IT security requirements, you can make this user a "sudoer", in other words enable this user to perform commands as the root user. MS Windows calls this "Runs as Administrator...".

We do not suggest altering the default permissions (or the umask) of the predefined unpriviledged account www-data. Instead it is convenient that the mdis user should be enabled to change file/directory permissions and file ownerships of files that belong to the "www-data" account.

This is necessary because the webserver process will perform code generation and will need write access to several distinct directories on the backend server.
Also, you probably want to use the mdis unixuser to edit any datafiles and photos uploaded through the browser, which belong to www-data by default. Last not least, file ownership details can interfere with version control (git).

Thus you should give both the operating system user mdis and the user www-data (unpriviledged user) write permissions to certain directories: backend/dis_templates/forms/, backend/dis_templates/models/, backend/forms/, backend/models/, src/forms/. User www-data is the owner of the webserver process or service and will need to write in these directories when mDIS users work with the template manager.

More about Directory Permissions

It is quite common to put files in these directories under version control (git). When you checkout files from version control, you do so via the command line, using a regular user account, e.g. mdis. This user must be able to overwrite files that www-data owns, and vice versa.

If common unix user- and group permissions are not enough, you can define such fine-grained file-access rules via Access Control Lists, ACLs.

This makes it possible to...:

do installation tasks on the host system and within the VirtualBox
do seamless rebuilds of forms and tables from the command line, within VirtualBox
work interactively with the Template-Manager

See the Sysadmin Documentation for details and operating-system specific issues (user management etc).

Advanced System setup and configuration

For more common tasks see the Sysadmin Documentation

WARNING

This page describes advanced tasks that require the mDIS developer to make changes to the mDIS source code. Anyone who does this should know how to use Git (opens new window) in order to restore the codebase back to a known good state from any misconfigurations or unwanted changes.

NodeJS (opens new window), the Javascript runtime, should be installed on an mDIS developer's workstation. The Node module @vue/cli (opens new window) can be installed locally or "globally", -g flag, for building the mDIS frontend.

Updating Third-party JS Code with `npm`

npm is three things:

npm is the Node Package Manager, a command line tool.
npm is a large software registry on the internet. See npmjs.com (opens new window).
(optional): A service on website npmjs.com which can grant you a login to manage your own software packages there. (We have not uploaded mDIS code there)

mDIS relies on the command line tool npm to fetch software from the npmjs.com registry. This is necessary during installation of the system. Occasionally it might become necessary during maintenance, e.g. to check for updates. Use npm to download and apply these updates.

The npm tool uses the package.json file in the base directory as a configuration file for the third-party code mDIS needs. It stores the metadata about what it actually downloaded and installed from npmjs in file package-lock.json.

Updating Third-party Code needed for Development: Client side

The update-commands as such are simple, but verifying that upgrades do not break anything can be quite time consuming. The command-Line tool npm, the Node Package Manager, will be used to perform the upgrades.

TBC

See also this section from the Sysadmin Documentation which describes server-side updates, in particular: other, less critical updates such as monthly security patches for the operating system, for example.

Update the client side on a development environment first, then create a production build with npm run build. If your mDIS instance runs on a different machine than your development machine: copy the contents of directory web/ to the production server directly. If you prefer to upgrade mDIS in a more careful way, perhaps put web/ on a staging server first, or set up a Continuous Integration Pipeline. It's up to you to control and fine-tune the upgrade process.

For the client side of a development machine, many dependencies are stored in directory node_modules/. In March 2020, there are over 1000 subdirectories in node_modules/, occupying ~340 MB of disk space. The nodejs runtime and its "global" dependencies reside in ~/.nvm/versions/node/v12.16.1/. Node will require an extra ~100 MB of storage space for the mDIS production deployment, and ~150 MB for development deployment because module vue-cli gets added.

Basically, upgrade the mDIS client side with

cd /var/www/dis     # where file "package.json" resides
npm list --depth 0  # show high-level structure of packages needed
npm outdated        # list upgradable third-party code
npm update          # update third party code (security updates and minor patches only)

1
2
3
4

Verification:
- TBC - a cumbersome task. Do this by trial and error 😉, and run test suites.

Command npm list --depth 0 shows the major Node/Javascript modules that mDIS relies on. In March 2019 there are 38 direct dependencies required. The 1122 directories mentioned above contain the dependencies in the subdirectories of these 38 direct dependencies. The node_modules/ directory structure still appears relatively flat, because this reduces redundancies; moreover, directory structures that are nested very deeply can cause problems with the operating system.

Command npm outdated will check the npm registry (opens new window) (docs) (opens new window) to see if any (or, specific) installed packages are currently outdated.

Command npm update will update all the packages listed to the latest version, respecting semantic versioning. Only "minor version upgrades" and "patch version upgrades" will be applied by npm update.

Run the command npm help update for details.

npm run build

In a production environment, only the essential Javascript dependencies necessary to run the front end are packaged together into directory web/. The Webpack bundler merges them into a few files that are highly optimized for disk space and performance. The most important file is called app.js and occupies only about 2 MB. Third-party code lazily loaded by app.js is stored in files called chunk_*.js.

The packaging is achieved by command npm run build.

At compile time, in a development environment, to build the mDIS frontend, an install size of ~ 400 MB of node modules is needed, but at runtime, the actual javascript code of the mDIS product, loaded by the browser, is ~3MB (publish size).

Again: For more common tasks and build-system details, see system setup - for SA's.

For a good explanation of Javascript dependencies for the "Gatsby" Software, read this blog post (opens new window).

Template Manager

General Usage

Using the GUI of the Template manager, the developer can create, update, duplicate, or delete Data Model Templates and then Forms Templates, in that order.

Note

Generating tables and forms is an essential DIS administration task.

This large topic has been placed into separate pages, to avoid that this page gets unreasonably long.

Advanced usage

For details, see

the separate Template manager page.
For data models templates JSON structure, see model JSON Files
For forms templates JSON structure, see form JSON Files
mDIS Forms: for basics, see the introduction page.

For the output of a single run of the Template manager, see section "Summary" below.

The Yii2 class hierarchy (PHP Files) explained.
See also a separate document on gii for a deep dive into code generation.

Code generation principles

Technical note

Most PHP code gets generated into directory backend/models. The details of the Code Generation mechanisms are hidden from mDIS administrators by a GUI and an infrastructure that is explained in simple terms below.

Internally, the mDIS Template manager consists of yii\base\Module classes, which make use of Gii (opens new window), the Yii2 extension for code generation. For background on Gii, see also a separate document.

For implementation details, see directories backend/modules/api/, backend/modules/cg/ and file backend/config/web.php.

These are the main software components that are involved on the server-side (backend) and on the client-side (frontend):

Backend

All this is explained in greater detail in the separate Template manager page and in this background document.

Model template generator output, directories and files:
- dis_templates/models/*.json (not shown in figure above): These files contain model definitions and model metadata. These files can be used to transfer table definitions from one mDIS instance to another (not the data, though).
- models/base/Base<model name>.php files. These are classes which inherit from models/base/Base.php. This class implements the extremely feature-rich Yii ActiveRecord (opens new window) functionality. models/base/Base<model name>.php classes are responsible for communicating with the Mysql database. These files will be updated (overwritten) every time you change and generate the data model template. Therefore, do not change the code inside any Base*.php files. (Of course, mDIS developers who know PHP may still change them temporarily, for debugging purposes for instance).
- models/<model name>.php files. These classes inherit from Base*.php classes. All the specializations you want to implement should be written in here. The models/<model name>.php files will not be accidentally overwritten during any subsequent code generation steps. This also means, the models/<model name>.php files get generated only once (unless you check "overwrite" in a certain modal dialog box that pops up when working with the Template manager GUI). Thus, keep them under version control (git), and manually roll back any unwanted changes.
Form template generator outputs:
- forms/<form name>Form.php files. These classes inherit from models/<model name>.php classes (the specialized one, not the base one). If there you need some customizations that belong only to this specific form, but not any other form that was derived from the same data model, they should be implemented in here.
- dis_templates/forms/*.json (not shown in figure above): These files contain form definitions and form metadata. These files can be used to transfer basic form definitions from one mDIS instance to another.

Frontend

Form template generator output, directories and files:
- The component DisSmartForm.vue is responsible for rendering the form on the fly, depending on the information in the template.
- Both SmartForm and customized forms use the DisForm.vue component internally.
- src/components/*.vue.generated files (not shown in figure above). These files are just a template, ready for copy-and-paste-and-rename. They are the starting-point for software developers, to implement customizations that belong to the client-side of a form. As long as these files have an extension other than .vue, they will be ignored by the mDIS application and the Webpack bundler (which mDIS relies on internally). In particular, .vue.generated files will not be used during the npm run build process, and will not be bundled with the mDIS application. In contrast, .vue files will be included in th build process.
- In case a customization is needed, remove the .generated extension, change the new .vue file according to your needs, and build the mDIS frontend to use the customized form. Do so with npm run build. Keep the new .vue file under version control (git).

Historical Perspective

For a comparison of code generation within the Legacy DIS app, see also an older (internal) document: "Code generation in the mdis app (opens new window)"

Summary

If you used the Templates-Manager to generate a new database-backed form, LithologyTestForm3, associated with a tableset Lithology, the mDIS code generator would create these 9 files:

backend/dis_templates/forms/lithology-test-form-3.json
backend/dis_templates/models/LithologyLithologyTestForm3.json
backend/forms/LithologyTestForm3Form.php
backend/forms/LithologyTestForm3FormSearch.php
backend/models/LithologyLithologyTestForm3.php
backend/models/LithologyLithologyTestForm3Search.php
backend/models/base/BaseLithologyLithologyTestForm3.php
backend/models/base/BaseLithologyLithologyTestForm3Search.php
src/forms/LithologyTestForm3Form.vue.generated

Most of these files belong to the mDIS server backend. See figure above.

The .json files (1,2) contain form definitions and table definitions. File (3) implements the main panels of the actual form. Files (5,7) implement the linking of the form to the database table. The *Search.php files (4,6,8) implement the Filter Bar and the "Filter by Values" Dialog. Only one file, the .vue.generated file (9), belongs to the Frontend, and customizing this file is optional.

Frontend

See figure above and DisForm Tutorial on how to customize .vue files. This is rarely needed, though.

For a detailed explanation of the structure of the PHP code generated and the class hierarchy, read this background document.

The Yii console runners

This topic pertains to system setup, system configuration and system testing.
Yii2 comes with a built-in console app that can customize the system after generating the initial bundles (mDIS backend from git, mDIS frontend with npm).

yii and yii.bat

In the main directory of the mDIS application, there is a small command-line program called yii. There is also a second program called yii.bat, which can be used on Windows.

These small console programs do not much more than loading a configuration file, and then delegate to the main console app which is located in backend/vendor/bin/yii.

This app is itself based on the Symfony Console component (opens new window). It is a powerful tool, and it is used by many other PHP frameworks, such as Laravel, Drupal, and TYPO3.

Open a shell and change into the directory at the root of the code base. Just run ./yii to get a glimpse of what the console app can do. It will output the command line switches and arguments, in a distinctive green/yellow color scheme.

Creating users and setting credentials with ./yii is one important task of many.

Also take a look at the configfile backend/config/console.php. This is the main configuration file for ./yii.

yii Migration Scripts

The main task of ./yii is calling the built-in migrate command. For details, run ./yii help migrate.

The migrate keyword is just a synonym for "custom script that safely modifies the mDIS backend or the mDIS database".
These migrations are mostly "database migrations", but they can also perform system maintenance tasks. Migrations have a certain behavior and a certain naming convention, and thus they are a mini-framework in themselves.
(The "migration" idiom was popularized (opens new window) by the web application framework Ruby on Rails (opens new window) since version 1.0, available circa 2005. Many other languages and frameworks have adopted the migration concept since then).

When you run yii run migrate or its shorthand yii migrate, Yii checks all PHP files inside the backend/migrations/ subdirectory, which are sorted naturally by the datetime stamps at the start of their filename, and executes the method safeUp() in all of those PHP classes. This happens alphabetically by filename. However the filenames contain the creation date of the migrations. Thus PHP applies migrations in fact chronologically, from first to last.

Thus a simple method of chaining dependencies is established. If a migration needs to be executed before another migration, it is enough to give it a filename that is alphanumerically sorted before that other migration.

Method safeUp() executes each migration as a database transaction. Theoretically, if the migration fails, the database is rolled back to its previous state. This is an important feature, because it allows you to run migrations on a production system without fear of breaking it. (However rollbacks are not implemented in mDIS an we rely on the database engine to do the right thing).

There is a whole lot more to learn about migrations.
See also the About migrations section in the Sysadmin page, and (optionally) the reference literature on Yii and Ruby on Rails to find more information.

For example:

The Definitive Guide to Yii 2.0 (opens new window)
Wikipedia on 'Migrations' (opens new window) - shows that the term is many times overloaded (similar to the term 'Model')
(optional) Ruby Migrations in the RubyOnRails Documentation (opens new window) - Etymology for the term "migration"

ICDP custom migrations

For mDIS we have created a few custom migrations.

These are the migrations we have in place. Most of them are only needed when a new mDIS needs to be set up and configured.

The most recent kmigrations are partly optional and running them is subject to the personal preference of the administrator.

#	Filename	Task (TODO add better descriptions
1	m180907_120508_init_rbac.php	init Role based Access Control (RBAC)
2	m180928_092529_add_api_token_field_to_user.php	add API token field to user
3	m181015_152554_copy_dis_templates.php	copy DIS templates
4	m181015_162413_list_values_table.php	list values table
5	m181015_162554_create_tables_from_templates.php	create tables from templates
6	m190218_144637_add_token_expire_field.php	add token expire field
7	m190709_135452_create_widgets_table.php	create widgets table
8	m190716_140314_create_post_table.php	create post table
9	m190717_122306_add_colors_to_widgets.php	add colors to widgets
10	m190719_133741_create_message_of_the_day_tables.php	create message of the day tables
11	m190809_124330_update_widget_to_be_clonable.php	update widget to be clonable
12	m200406_130153_add_config_table.php	add config table
13	m201030_115123_create_files_from_templates.php	create files from templates
14	m201124_112123_change_value_list_table.php	change value list table
15	m210111_141330_split_change_still_exists.php	split change still exists
16	m210605_214919_add_ldap_field_to_user.php	add LDAP field to user table
17	m211009_103700_create_igsn_table.php	create IGSN table
18	m211130_140713_fix_user_config_index.php	fix user config index
19	m220202_142138_revise_rbac_tables_for_sql_server.php	revise RBAC tables for SQL Server
20	m220227_080122_create_session_table.php	create session table
21	m220306_193822_seed_value_lists.php	seed value lists
22	m220601_182151_remove_parent_from_sample_request.php	remove parent from sample request
23	m220824_103348_add_samples_cvs_widget.php	add samples cvs widget
24	m220829_073951_add_default_user_and_its_role.php	add default user and its role
25	m221102_132954_add_moratorium_period_guard_rule.php	add moratorium period guard rule
26	m221122_100831_change_administrator_role_name_to_administrator.php	change administrator role name to administrator

(Table is Output of ls -1 m* | perl -nl -E "\$f = \$_; s/^m\d+_\d+_//; s/\\.php//; s/_/ /g; say qq(\$f, \$_)" | csvlook --no-header -l), plus a few manual edits

Migrations have state

To see how migrations are carried out, and the current state of previous migrations, check the backend/config/console.php configuration script (section controllerMap => [...]), and in particular see the backend/commands/DisMigrateControllers PHP file. See also the migrations table in the mySQL database of your mDIS application.

Apply migrations only carefully in a "nonempty" mDIS

Migrations are designed to be run on an "empty" mDIS. They should be run when mDIS is initially installed.

Actually, when you run mDIS in production, the *.json files in directory backend/dis_templates/ have higher precedence than data definitions emitted by these (old) migration scripts. This means there is some risk overwrite your own, carefully crafted JSON files with unwanted output from these migration scripts!
Usually, the JSON configuration files hold more recent and important information, e.g. the exact form designs, which are not available during system setup (because it is You who customizes and designs them).

The mDIS code generator will usually, but not always, look into this directory. It may ignore everything else (e.g. SQL create table statements given in .sql files), and it will give precedence to the information in the JSON templates.

TODO: clarify this.

mDIS MVC Architecture

Yii2 is an MVC framework. As such it is useful to explain what these terms mean with respect to mDIS.

The term "Controller" mentioned below and in the sidebar refers to the Model–View–Controller (opens new window) (MVC) idiom of software architecture. A controller class accepts input and converts it to commands for thedomain models (here, database tables) or views (here, webpages).

mDIS Models

The code for most domain models gets generated with an interactive model designer. That is the point of the sections "Template manager" and "Code generation principles" mentioned above. The PHP files for linked tables of science data reside in directory backend/models.

mDIS Views

There are very few distinct MVC views rendered by mDIS (actually by the PHP Yii framework), because most webpages get rendered by an "independent" JavaScript-based frontend. See the figure above. Specifically, that non-MVC mDIS frontend is based on the VueJS framework, see below and DisForm.vue. The frontends make REST API calls to fetch the data that they need.

The MVC View code for the login-page resides in backend/views/site/index.php.
Headers for mDIS Reports, and simple User-Management Pages can also be found in backend/views/site/.

TBC

mDIS Controllers

The most important mDIS controllers are where the REST API implementation resides. That is in directories backend/modules/api/common/controllers, backend/modules/api/modules/v1/controllers, and backend/modules/cg/controllers.
Furthermore there are controllers for basic tasks, e.g. rendering the login page, a controller to render the imported Archive-Files, and for rendering the Pull-Down Menu with the List of Exportable Reports. These controllers are in backend/controllers.

TBC

REST API

The mDIS web application uses a REST (opens new window) programming interface to communicate between frontend and backend. The API could be used to create new types of forms, with a different look and feel, for instance. But the REST API is also available to external developers.

mDIS actually has two REST APIs: a "regular", REST API for science data querying and manipulation, and the code-generation API (informally named cg-API) for data definition operations such as "create table", "create domain model", etc. The cg-REST-API is for internal use and, therefore, less exposed and more advanced. Only mDIS experts should use it.

The "regular" REST API implementation resides in directory backend/modules/api/modules/v1/controllers, and the cg API in backend/modules/cg/controllers.

See REST API page for extensive documentation. For tutorials, see Worked Examples.

These tutorials do not demonstrate how to edit mDIS datasets (create, delete, duplicate, etc) via the mDIS REST API. These tutorials focus on the simple case of getting data in a read-only mode.
Such data-modifying API calls are available, though. If you need to learn to get write-access to the mDIS REST API, study the JS code in the dis-data-gen repository (opens new window).

Worked Examples

Most calls to the mDIS REST API are password protected. To skip password requests, you need to login only once and get an authorization token called a Bearer-Token (opens new window). This security token looks like a long password, e.g. DssMim-2QERdh6DcxPV0Saj9qYuKo267, and it works like a session-id or HTTP cookie, however it is not stored in a Cookie:-Header but in a different HTTP header field, and in Local/Session Storage, optionally.

Browser Console

Run this in the console of your browser.
The console is part of what is know as Developer Tools. Hit F12 key (or Ctrl+Shift+i) to open the developer tools. Then select the "Console" tab.

Here is a code snippet that shows how to get a Bearer-Token from the mDIS REST API.
You can copy and paste this into a browser console, and run it as-is. Important: For this example to work, your browser tab must have any page open at data.icdp-online.org, to avoid CORS errors (Cross-origin resource sharing).

////// Part 1 /////////////////////////////////////////////
    // Start of login, do this only once
    let mdis_url = "https://data.icdp-online.org/mdis/demo";
    let logindata = {};
    [logindata.username, logindata.password] = prompt(
        "Enter username and password, separated by blank:",
        "user2 user2_password"
    ).split(" ");

    // Get Bearer token, save to Session Storage
    let response = await fetch(`${mdis_url}/api/v1/auth/login`, {
        method: "POST",
        headers: {
            "Content-Type": "application/json",
        },
        body: JSON.stringify(logindata),
    });
    let data = await response.json();

    sessionStorage.setItem("token", data.token);
    // Set authorization header
    const myHeaders = new Headers();
    myHeaders.append("Content-Type", "application/json");
    myHeaders.set('Authorization', `Bearer ${sessionStorage.getItem("token")}`);

///////////////////////////////// end of Part 1, login  //////////////////////////

// Part 2: ### Get data from mDIS, here: coreboxes
let url = `${mdis_url}/api/v1/form?name=corebox&per-page=5&page=1&sort=id&`;
let response2 = await fetch(url, {
  method: "GET",
  headers: myHeaders,
});
let data2 = await response2.json();
console.log(data2.items);

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

The example above prints the first 5 coreboxes to the browser console. No filtering is applied.

Unwanted Logouts

Experimenting with the mDIS REST API might log you out of your "normal" mDIS session.

There can only be one Bearer-Token active per user at any given time. If a user logs in twice with a new token (and not the token already gathered), the web server will reject the older token from the previous session, and only accept the newer token. This also holds for "regular" browser windows with open mDIS sessions.

Therefore, if possible,

do not use the same login from two different browsers on the same computer at the same time
do not use the same login from two different devices simultaneously (e.g. your Laptop and your Smartphone)
use a test account with its own credentials, rather than your personal account, for your scripts calling the mDIS REST API. Then, at the same time, you can use your personal account for your mDIS browser window without being logged out.

JavaScript

There is a set of more advanced tutorials for using the mDIS REST API with JavaScript with NodeJS. How to render your own HTML table, how to perform low-level (old-school) asynchronous access.

ICDP internal: Not a real tutorial, but there exists a "Postman collection" which is also based on JavaScript (and NodeJS).

R

There is a brief tutorial for R. Fetch some data from the mDIS REST API, perform a few exploratory analyses.

bash

This example uses Unix command line tools to get some data from the mDIS REST API:

curl (opens new window) to perform web requests, and
jq (opens new window) to parse the JSON response from mDIS.

Powershell

Powershell 6 script: for quick and dirty command-line processing on Windows.

Excel, Libreoffice-Calc TBC

There are two scenarios:

Import from within Excel, from within an empty spreadsheet by connecting to mDIS, via an Excel-Extension or VBA.
Creating an Excel Sheet with an external program, filling it with science data, and opening it with Excel. See Powershell example above. Take a close look at the commented-out lines.

TBC

Google Sheets TBC

There are two scenarios:

Import from within a Google Sheet by connecting to mDIS, via Google Apps Script
Pushing/Exporting to a new Google Sheet from some other software, or from the command line

TBC

Perl, Python, Go, Java, ..., PHP

Coming Soon

really

DisForm.vue

The role of the DisSmartForm and DisForm components was shown in the figure "Code Generation Principles/Frontend" above. DisForm is a powerful component that establishes the basis for all data-input forms.

See DisForm.vue for more background and how-to-use instructions.

Calculated Fields

Calculated fields on the client side can be created with a feature in the Template Manager. A calculation formula should contain no more than a single line of Javascript code.

Reminder

See "For Operators/Viewers" what Calculated Fields are and how to perform short one-line calculation expressions.

Calculated Fields should not be set to "required" in the form designer. This can create problems when saving records.

There is also an extensive tutorial on how to create more complex calculated fields in input-forms. This is needed when the calculation procedure is longer than a one-liner.

Keyboard Shortcuts

Defining a new custom keyboard shortcut requires:

Adding a custom shortcut to the small file src/util/shotrcuts.js - e.g. Shift+Alt+r
Adding a new method to DisDataTable.vue that implements the shortcut action or event handler.
Binding the keyboard shortcut to the dis-data-table Vue component : v-mousetrap="{keys: [shortcuts.focusOnTable, shortcuts.scrollBottomRight], in DisDataTable.vue.

Form Validation

Server-Side

On the server-side, form-data validation happens when a user attempts to save a record.

Validators are stored in backend/models/base/Base*.php files. Check the code of the rules() method in each class. The mDIS validators work declaratively by inheriting from the many built-in Yii input validators (opens new window) in the backend/vendor/yiisoft/yii2/validators directory.

Do not change rules() methods in any PHP-file of the backend/models/base/ directory. You can change the PHP-files containing the derived classes in the backend/models/ directory. Some files in that directory already provide their own rules() methods, for instance ImageOfTheDay.php, ListValues.php, ListValuesSearch.php, MessageOfTheDay.php, Post.php, Widgets.php. Take a look at them for inspiration.

For example, in backend/models/core/DisListItem.php the validation rules are declared like this:

  public function rules()
    {
        return [
            [['listname', 'display'], 'required'],
            [['sort'], 'integer'],
            [['listname'], 'string', 'max' =>  50],
            [['display'], 'string',  'max' => 100],
            [['remark'], 'string',   'max' => 255],
            [['listname', 'display'], 'unique', 'targetAttribute' => ['listname', 'display']],
        ];
    }

1
2
3
4
5
6
7
8
9
10
11

Method rules() returns an array of arrays. The first subarray contains the field names that are required (here 'listname' and 'display').

These properties correspond to SQL column constraints and index definitions.

TBC

The rules() method might have some additional value, 'safe'. No attribute is marked 'safe' above. But see backend/models/base/BaseProjectProgramSearch.php:

     public function rules()
    {
        return [
            [['id', 'name', 'program', 'remarks'],'safe'],
        ];
    }

1
2
3
4
5
6
7

This means this attribute is "whitelisted". The attribute is not constrained by any validation rules and still marked as 'safe'. Users can assign anything to this field. If it was not marked as safe, it would be ignored by the setAttributes() method of the yii\base\Model class.
Attributes known to be _Un_safe must be whitelisted with '!', e.g. "!last_command". (We do not use this feature in 2019).

Calculated Fields (server-side)

The transformation of the formula entered in "Calculate" to JavaScript is performed by the PHP method getJsCalculate() in File backend/components/templates/FormTemplateField.php and Javascript in Vue Files src/components/DisSmartForm.vue and src/components/DisForm.vue.

For details, see Form Validation - Calculated Fields examples.

Yii Behaviors

Yii Behaviors (opens new window) allow to implement functionality outside of a data table. Yii Behaviors are implemented in PHP code. They are related to mixins or PHP traits (opens new window).

In mDIS, Behaviors are used to create complex constraints and calculated fields. Similar to database triggers, Yii Behaviors can be bound to many different events on a data table. Usually they are bound to insert- and update- Events.

mDIS makes use of Yii Behaviors to ...

concatenate column-values in a very specific way (as "combined keys", e.g. "5063_1_A_3_1")
create IGSNs (opens new window)
create versatile autoincremented values that can be customized (e.g. they can have gaps, can be nonunique, can be given a start value > 1)
create numeric multi-row constraints on some groupings inside a data table (ChildrenLimitBehavior)
carry-over state and logic from parent-entities to child- entities (e.g. DefaultFromParentBehavior)
and more.

The mDIS Template Manager comes with a Graphical User Interface (GUI) to set some parameters for these behaviors, and to attach them to tables and models. See the table creation page of the Template manager.

TBC

(Unfortunately the documentation for the GUI for the template behaviors is still lacking at this time.)

TBC

However it is still very likely that you need to change the mDIS source code to create or adapt the behaviors you require, to manage the specifics of your geological materials and science data. Thus you probably need access to the generated source code for the model and for the derived class. The code is in backend/models/Base*.php files.

Intermediate PHP programming skills needed

Creating or adapting existing Yii Behaviors requires changing the PHP source code of mDIS. This entails opening .php files, changing some code, saving the files, reloading web page. (A compilation step is not required).

See Customizing models and templates with Yii Behaviors for details.

File Importers

Types of Importers

File Importers are available through the "File Upload" feature.

For their implementation in PHP, see the directory /backend/importers. It contains files implementing this inheritance hierarchy:

Base.php                    # abstract class, contains 11 methods
  CsvImporter.php           # contains 13 methods
    ExampleCsvImporter.php  # contains specialized beforeRun(), readRow()
  ListValuesImporter.php

1
2
3
4

For the controller class, see backend/controllers/ImporterController.php.

The controller searches for Importers in directory backend/importers. They must end with .*Importer.php. Every importer inherits (via backend/importers/Base.php) from yii\web\ViewAction and (therefore) is run automatically by the Yii framework.

CSV Importer

There exists one "universal" CSV importer, backend/importers/CsvImporter.php.

For each importer, specify different file extensions of the data files to be imported
Ignoring columns in the import file, if necessary
Renaming (and remapping) columns of input files during import
Add extra columns that are used to calculcate other columns in the target table
Set model/table attributes to fixed values
Check and Correct the imported data values
Determine the parent record (Foreign Key) in a custom way
Target Table/ "data model" does not have to be selected, since it is fixed or determined by the import filename extension (?)

There is a short file, backend/importers/ExampleCsvImporter.php, that shows how to specialize the universal CSV importer. It sets some default values specifically for attributes of the core_core table. Details are not shown here.

Based on these two production code examples of importers it should not be too difficult to implement further importers for different formats.

Hints and Limitations

Adding new file-importers requires custom PHP programming. But usually it is only a single file that needs to be edited. Often a simple workflow of copy, paste, adapt might goes a long way (especially fort one-off import tasks).

Specifying foreign keys must happen (using the Template Manager) before a datafile import starts. Foreign Keys cannot be added easily to a nonempty table through the Templates-Manager GUI. Preexisting dependent tables must be emptied first, in order to avoid foreign key conflicts or collisions.

For Importer usage see also "For Operators/Viewers" (basic use) and another short section (optional).

ListValues Importer

The list values files must be simple text files, more specifically: semicolon-separated values.

TBC

Reports

Reminder

As mentioned in the Beginner documention, there are single-item reports and multi-item reports.

Report items

Storage Locations of graphical items in the Backend

Many reports contain graphical items such as logos, headers, footers, formatting directives, etc. Where are these items stored?

Answer: Internally, some reports use Yii Asset Bundles (opens new window). AssetBundle represents a collection of asset files, such as CSS files, JS files, images. They are stored in directory backend/assets.

Other HTML-, CSS- and JS-code fragments are stored as Here-Document Blocks (opens new window) inside some methods of some PHP files such as /backend/reports/SectionMarumQrCodeReport.php. From there they are returned by methods such as getJS().

Customizing Reports

Backend: Change the PHP files in directory /backend/reports/. (Continued below).

Frontend: Any client-side changes are unnecessary. DisForm.vue has code that dynamically filters and loads any reports available from that form. The files are loaded automatically from /backend/reports/. For illustration, this is the code snippet that loads all single-item reports of a form, when you click on the upper "Export" button, and then displays a little pulldown menu with available reports:

<v-list v-if="selectedItem && selectedItem.id">
  <v-list-tile
     v-for="(item, index) in reports.single"
     :key="index"
     :href="`/report/${item.name}?model=${dataModel}&id=${selectedItem.id}`"
     target="_blank"
     >
    <v-list-tile-title>{{ item.title }}</v-list-tile-title>
  </v-list-tile>
</v-list>

1
2
3
4
5
6
7
8
9
10

The code for displaying the pulldown menu of multi-item reports looks very similar.

Backend (implementation details)

File backend/controllers/ReportController.php is in charge of processing the :href=/report/ URL-path-fragment in the code sample above. The controller is configured in /backend/config/web.php. It looks for PHP files in directory /backend/reports following this naming convention:

$reportsPath . $reportName . "Report.php"

Thus, filenames of mDIS reports must end with Report.php (similar to importers mentioned above).

PHP classes of mDIS reports must be named/namespaced according to this rule:

"\app\reports\" . $reportName . "Report";

Inside any such class, it must have:

a constant property MODEL which specifies on which form's pull-down-menu the linked report will appear, and
a property SINGLE_RECORD specifying if it is a single-item report or a multi-item report:

class CoreQrCodeReport extends Base
{
    /**
     * This report can only be applied to Core forms.
     */
    const MODEL = 'CoreCore';

    /**
     * This report prints labels for ALL filtered records of the form.
     */
    const SINGLE_RECORD = false;

    const TITLE = 'Print Core QR code';
    // more code to generate CSS/JS/HTML fragments, not shown
    function getTemplate(){}
    function getLabelTemplate(){}
    function getJS(){}

    function generate(){} // return the whole report
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

The PHP class above contains a few methods which will generate and return the report's content.
For brevity, only empty "function stubs" of these methods are displayed here (the empty {} pairs). In reality, these are not empty.

continued in : For Developers - ICDP internal

mDIS For Developers

Related Pages

Roles and Permissions

Operating System: Users, Permissions

Advanced System setup and configuration

Updating Third-party JS Code with npm

Updating Third-party Code needed for Development: Client side

Template Manager

General Usage

Advanced usage

Code generation principles

Backend

Frontend

Summary

The Yii console runners

yii and yii.bat

yii Migration Scripts

ICDP custom migrations

Migrations have state

mDIS MVC Architecture

mDIS Models

mDIS Views

mDIS Controllers

REST API

Worked Examples

Browser Console

JavaScript

R

bash

Powershell

Excel, Libreoffice-Calc TBC

Google Sheets TBC

Perl, Python, Go, Java, ..., PHP

DisForm.vue

Calculated Fields

Keyboard Shortcuts

Form Validation

Server-Side

Calculated Fields (server-side)

Yii Behaviors

File Importers

Types of Importers

CSV Importer

ListValues Importer

Reports

Report items

Storage Locations of graphical items in the Backend

Customizing Reports

Backend (implementation details)

Updating Third-party JS Code with `npm`