mDIS For Developers (ICDP internal)

Related Pages

mDIS For Developers

ICDP internal

This section is somewhat specific to ICDP internal mDIS installations. However, you might still find some pearls of wisdom here on this page.

Testsuites

In 2020, mDIS has grown in complexity, so automated software testing has become more important.

Note

This is also a system administration task.
You'll need shell access to run most of these tests, or access to the Continuous Integration infrastructure, or be in physical possession of the Postman suite (described below).

Of course, you need valid mDIS credentials, too.

Backend Unit Tests

• The Yii developers recommend the end-to-end testing tool Codeception. It is installed in mDIS' backend/vendor directory because it is a third-party code dev-dependency of Yii2.
  To run the Codeception tests:
  • run php backend/vendor/bin/codecept run unit.
  • run php backend/vendor/bin/codecept run functional or
  • run all tests, create coverage reports, and show the results on the command line:
    ./backend/vendor/bin/codecept run --coverage-text --coverage-html --no-colors To enable the codecept test runner, you must:
  • set up a test database dis_test (or similar) with some tables inside that match the real mDIS schema.
  • declare the following environment variables: MYSQL_PASSWORD, MYSQL_DATABASE, MYSQL_USER, MYSQL_HOST with appropriate values. Pro tip: create a .env file such that you can simply run . .env.

REST-API Tests

• For testing the REST API of the mDIS PHP backend, there exist ICDP-internal testsuites, available as collections for the Postman test runner. The testsuites contain many tests of API calls provided by the mDIS PHP controllers. Postman testing automates the following actions: create items, insert, update, delete, login/logout, and, of course, many different variants of GET operations.
  This collection is available to all ICDP team members.

User-Interface (GUI) Tests

• For testing the mDIS User Interface, ICDP Data Management has developed a few scripts for running GUI tests with Codeception and Selenium. However, they are quite brittle. They run only in a very specific configuration; and even then, they run slowly. Ask us for access or assistance with these test scripts.

• Optional. There are a few unit tests for JavaScript code, currently for Datetime entry only. Run testsuites on the command line with

  • npm run test:unit or
  • npm run test:unit -- --watchAll
    (This is just a basic example without much test coverage.)

Fixing problems with the test environment

Codeception allows for Unit tests, Functional tests, Integration Tests, Acceptance Tests, GUI tests - all in one environment. This infrastructure encompssaes the testrunner, and for example, a way to link a browser to run GUI Tests: and more.

Therefore the Codeception test environment is quite tricky to set up. Also, the testsuite for functional tests needs an empty physical database to be present.

If you have testrunners that cannot start, or tests that fail, then either:

• something is wrong with your Codeception installation
• something is wrong with your mDIS installation
• the schema of the test database dis_test, which the Codeception test runner uses for a few generic tests, is different from the schema your database uses.
• the mDIS codebase has evolved, but the tests have not been updated, or:
• you have customized mDIS such that new forms, models, or tables have been created or altered. Thus, tests that assume a "vanilla" (= default) installation fail.

In any of these cases, try to find out yourself what has happened, in a step-by-step manner.

To do that, run single tests, and/or run the tests with more verbosity.
To run single tests, perhaps do this:

Run unit tests only:

php backend/vendor/bin/codecept run unit backend/tests/unit/templates/models/FilesUploadedFormTest.php

To run tests with more verbosity, add -vv or vvv to the command line.

php backend/vendor/bin/codecept run functional -vv

Debugging and Introspection

There are various ways to look under the hood of a running mDIS. Here are some of them, debugging and introspection.

Step-Debugging PHP with XDebug 3

Debugging the mDIS Backend

Step-Debugging means going through running code line-by-line.
Doing this with the free PHP debugger "XDebug" is documented in PHP Debugging.

Debugging the mDIS VueJS Frontend

Debugging and hot-reloading of the VueJS frontend is important for developers.

There are numerous ways to start a debugging session for VueJS Apps. Here is my current favorite. It is a bit specific to the VSCode-Insiders IDE on Linux:

Debugging the mDIS VueJS Frontend

Alternative: Attach to Running Process
If you prefer to start your app manually:

• Open a new terminal window.
• Start your app in serve mode with npm run serve. Check the output. It should look like this:

npm run serve

> dis@3.0.0 serve
> vue-cli-service serve

  App running at:
  - Local:   http://localhost:8080 

The app is served on port 8080 by node.js.

Inside VSCode-Insiders, add this configuration to your .vscode/launch.json file:

{
    "configurations": [
      {
        "type": "chrome",
        "request": "launch",
        "name": "Manually: Attach to Vue",
        "url": "http://localhost:8080",
        "webRoot": "${workspaceFolder}",
        "breakOnLoad": true,
        "sourceMapPathOverrides": {
          "webpack:///src/*": "${webRoot}/src/*"
        }
      }
    ],
  }

Inside VSCode, inside the debug panel, select "Attach to Vue" from the debug dropdown and press F5.
VSCode will launch Chrome and connect the debugger.

Your VSCode will now pause at any debugger statement in App.vue (or any other .vue or .js file), allowing you to inspect variables and step through your code directly from within the editor.

Assumptions:

• Prerequisites Setup as just described above (VSCode-Insiders, npm, VueJS, Chrome)
• JavaScript Debugger Extension (Nightly)

Debugging Frontend while Running a Backend Remotely:

Often you have a running mDIS instance filled with data, but you want to develop the frontend independently.
When you have a running mDIS instance on a remote server, you can connect that Backend to your local development machine, where a development instance of the Frontend is running.

Consider this setup

• You have a running mDIS instance based on a VirtualBox VM, or Docker, or similar.
• You have PortForwarding for HTTP requests set up in VirtualBox, so you can access that mDIS instance from your local machine. Assuming the forwarded port is 8888, you can access the "remote" mDIS backend on your local machine with http://localhost:8888.
• Optional: Mount it to your local machine with sshfs or smbmount, or VirtualBox's shared folders.
• Change file vue.config.js at the following lines such that the devServer section matches your REST API endpoint:

module.exports = {
  //...

  // Webpack will proxy/forward all requests starting with /api to the mDIS backend.
  // This way you can debug the frontend on localhost, but the backend will run elsewhere.
  devServer: {
    // where node.js serves the frontend 
    // (will appear in output of "npm run serve" command)
    port: 8080, 
    // where the backend runs. Maybe 8000, see vue.config.js:
    proxy: {
      '/api': {
        target: 'https://localhost:8888',
        secure: false, // optional, set to false if you're using a self-signed certificate
        changeOrigin: true // optional, helps with virtual hosted sites
      },
      // some logos and images need to be proxyed, too
      '/img': {
        target: 'https://localhost:8888',
        secure: false, // optional, set to false if you're using a self-signed certificate
        changeOrigin: true // optional, helps with virtual hosted sites
      }
    }
  },
}

Introspecting (not step-debugging) VueJS Apps - The VueJS DevTools

Install the Browser Extension "VueJS DevTools (legacy)" from the Chrome Web Store.
These tools do not enable "real" step-by-step debugging. Instead, they offer introspection features:

• Component Inspection: Allows you to navigate the entire component tree and see the hierarchy of your Vue application
• Component State Examination: Lets you inspect and modify component data, computed properties, and props in real-time
• Vuex Store Debugging: Provides visualization and time-travel debugging for Vuex state management
• Event Timeline: Shows events as they're emitted
• Performance Profiling: Helps identify performance bottlenecks in component rendering
• Custom Routing Tab: Displays route information if you're using Vue Router
• Custom Inspector: Allows third-party plugins to add their own custom inspection panels

See this hint from our "VueJS DevTools" article.

The Yii2 Debug Bar

PHP and SQL introspection (not step-debugging) is possible with the Yii2 Debug Bar.

Intercepting SQL Statements processed by Yii

The Yii Debug Bar is a tool that intercepts HTTP requests, PHP workloads, SQL statements processed by Yii. You will see that a lot of permissions checks are done by Yii, and which SQL statements are sent to the database server (you need to change the logging settings to do so, see backend/config/web.php). For each SQL query fetching science data, there are 10 times as many user rights/permission checks. This is a lot of overhead, but it is necessary to ensure that users only see data they are allowed to see.

// example for configuring a SQL logger in Yii2
  'log' => [
    'traceLevel' => YII_DEBUG ? 3 : 0,
    'flushInterval' => 1,
    'targets' => [
      [
        'class' => 'yii\log\FileTarget',
        'levels' => ['info' ],
        'logVars' =>  ['!_GET', '!_POST', '!_FILES', '!_COOKIE', '!_SESSION', '!_SERVER'],
        'categories' => ['yii\db\Command::query'],
        'except' => ['information_schema.*',  'app_config', 'mdis_dive22.auth_item', 'mdis_dive22.auth_item_child', 'mdis_dive22.dis_session'],
        'logFile' => '@app/runtime/logs/sql.log',
        'prefix' => function ($message) {
          return $message[0]; // only log the message itself
         }
      ],
    ],
  ]

Getting Diagnostic information about PHP <-> Database interaction

The Yii Debug Bar is usually invisible. If it is enabled (by setting the environment variable YII_DEBUG = true), it will become visible on the mDIS Users-Manager page, as shown below in the screenshot.

It looks like this, and users must click to expand it from a minimized, collapsed state:

Expanded it looks like this:

Click on any field to expand the debug bar to an even larger debug window. (Not shown here).

Then you can search for many things, e.g., stack traces, error texts, and even SQL statements sent by Yii from the web server to the MySQL database.

To enable the debug bar, open a browser window with the official instructions and read it. Then follow along and do this:

Open file backend/config/web.php. Change this line of code

'bootstrap' => ['log', 'api', 'cg'],

to this:

'bootstrap' => ['log', 'api', 'cg', 'debug'],

This line of code is located at the top of the file (in line 11 or so).

Add this block of code to the modules config section in web.php:

// these are just code snippets, not the fuill file
// 'modules' =>  ,,,
  // ...
  'debug' => [
      'class' => 'yii\debug\Module',
      'allowedIPs' => [env('EXTERNAL_ALLOWED_IP'), '127.0.0.1', '::1']
  ],
  // another sql logger
  'targets' => [
    [
      'class' => 'yii\log\FileTarget',
      'levels' => ['info' ],
      'logVars' =>  ['!_GET', '!_POST', '!_FILES', '!_COOKIE', '!_SESSION', '!_SERVER'],
      'categories' => ['yii\db\Command::query'],
      'except' => ['information_schema.*',  'app_config', 'mdis_dive22
          auth_item', 'mdis_dive22.auth_item_child', '
          mdis_dive22.dis_session'],
      'logFile' => '@app/runtime/logs/sql.log',
        'prefix' => function ($message) {
              return $message[0]; // only log the message itself
        },
    ],
    // ...
  ]

Set the following variables in file (project_root)/.env, or set them manually inside web/index.php:

    YII_DEBUG=true
    YII_ENV=dev
    EXTERNAL_ALLOWED_IP=111.22.*.* # your IP address range

Troubleshooting the debug bar

If you still cannot see the Yii debug bar, then you should check if Yii has started collecting the debugging info anyway. Maybe Yii just cannot display the Debug Bar GUI Widget for some reason.
Check the backend/runtime/debug/ directory and have a look at the newest *.data files. These are text files in PHP's serialization format. They are not convenient to read, but you still can use them to find out what is happening. The grep tool can go a long way with this.

See also "Docker Logfiles" below.

Optional (?):

As an alternative to the debug bar, install Apache's mod_security2 module. This allows you to trace the payload of PUT and POST requests as they arrive in the container.

Do not enable the Debug Bar feature in production. It is probably bad for performance. In mDIS, you need to have administrator privileges to see it (and understand it).

Remove older sess_* files from the directory specified in PHP's session.save_path, e.g., /var/lib/php/sessions. (I don't remember when and why this helped me once).

SQL Debugging on MySQL

A simple way to find long-running SQL queries is to use the query SELECT * FROM sys.statement_analysis LIMIT 10, or work directly with the tables in the database performance_schema. You need MySQL 8 (or a MariaDB equivalent) for this, and your user needs access to the performance_schema and sys databases.

If you are using a MySQL variant as a backend database for mDIS, then you can also use the CLI tool mysqldumpslow to analyze all slow SQL statements issued to the database server. The mysqldumpslow command line client is part of the mysql package. It is installed by default, but it is not enabled by default.

-- SQL: Verify if Slow Query Log has been configured
show global variables like '%slow%';

You must enable it by adding a slow_query_log line in the my.cnf file. Then you must restart the MySQL server. Then you can use the mysqldumpslow command line client to analyze the slow SQL statements. You will get the SQL text of slow-running queries, and you will get a summary of the slow queries, sorted by the number of times they were executed, or by the total time they took, and which user issued them.
You should try this first on a local development machine, especially if you have more than the mDIS database on your local MySQL server. You might easily write much more text to the log file than intended.
A more detailed explanation is out of the scope of this article. But you can find some instructions on the Internet. //To be continued...//

Docker: Tips and Tricks

Getting into an mDIS Docker Container

Enter a running Docker container (the server-side, the backend) with:

docker exec -it disapp_php_1 bash  # run bash inside container

Docker Logfiles

Logfiles in the Docker Container

On the docker host, display the Apache Error Log of a running container with

docker logs -f <containername>

Although Apache log files are in the container's /var/log/apache2/ directory, they are symlinked to stderr, and therefore are not displayable from inside the container.

Logfiles inside the container

Logfiles the mDIS instance generates when used.

Viewing mDIS webserver- and PHP log files on Linux (inside a VirtualBox instance or inside a Docker container):

cd /var/www/dis/backend/runtime/logs
tail app.log  # mDIS yellow toast-boxes, error log
tail convert.log # During picture conversion, ImageMagick writes to this logfile.
# you can activate more loggers ! 
# See in the Yii2 configuration file `config/web.php`.
# Ensure to configure log levels and targets appropriately.

Accessible from outside the Docker container, too, when /var/www/dis is mounted as a Docker volume.

mDIS Documentation Tips

Offline reading

Download PDF Version for a comprehensive overview, enabling you to read the documentation offline in a single file.

Advanced: git-clone the GitLab repository, and read *.md files opened from your local directory. These are text files written in Markdown. The .md files are human-readable.

Optional, for developers: To generate HTML files locally on your computer, install Vuepress with npm i -g vuepress, and inside the repository's directory, run vuepress build. Then perhaps run one of the following commands to open the documentation as a web page in your browser:

• Python 3: python3 -m http.server 8808 --bind 127.0.0.1 > /tmp/simple-web-server.log 2>&1 &
• Python 2: python -m SimpleHTTPServer 8000; firefox localhost:8000
• PHP: php -S localhost:8000; firefox localhost:8000 Doc Page
• JavaScript/NodeJS:
  npx lite-server; (by default listens on port 3000, so try google-chrome localhost:3000)

These commands start small web servers that are built into the respective language interpreters (Python/PHP), or download a small web server lite-server from the internet (NodeJS). You might need to install lite-server with npm i -g lite-server (global) or npm i -g lite-server --save-dev (local).

See also: Big list of HTTP static server one-liners

Searching this Documentation

See the text field with the looking-glass icon [🔍], located in the title bar at the top of this webpage.

Searching with Vuepress

There is a search engine that comes built-in with Vuepress, but it is not very powerful. It only searches headlines and keywords that appear in the sidebar.

We have replaced Vuepress search with an index provided by the commercial indexing service Algolia. They have a free tier that we are using.

DEPRECATED since 2025: Searching with Algolia.com

Algolia-based search supports full-text search, similarity search, and it has a nice user pull-down interface.

The documentation site must be indexed with their scraper and pushed to algolia.com.

The indexing script:

#!/bin/sh
cd /home/knb/code/git/dis-docs/_work/dis-search/
# on wb45, the actual command to run the algolia indexer.
# details are in the .env file and prod_mdis.json file
docker run -it --env-file=.env -e "CONFIG=$(cat /home/knb/code/git/dis-docs/_work/dis-search/config/prod_mdis.json | jq -r tostring)" algolia/docsearch-scraper
cd -

Config file /home/knb/code/git/dis-docs/_work/dis-search/config/prod_mdis.json for the Algolia scraper + indexer:

{
  "index_name": "prod_mdis",
  "start_urls": ["https://data.icdp-online.org/mdis-docs/guide/"],
  "selectors": {
    "lvl0": "#app main.page h2",
    "lvl1": "#app main.page h2",
    "lvl2": "#app main.page h3",
    "lvl3": "#app main.page h4",
    "lvl4": "#app main.page table",
    "text": "#app main.page p,#app main.page ul,#app main.page ol, #app main.page td"
  }
}

Tips for Algolia

If the search query yields a 404 error, remove the duplicated section from the URL search result page. Often there is a duplicated path fragment in there. Change the URL fragment from mdis-docs/mdis-docs/ to mdis-docs/ and reload the 404 page.

Run the indexing script again. It must be run from within the directory dis-docs/_work/dis-search/ to create correct URLs.

The .env file mentioned in the script contains an Algolia APP-ID and an API key to give access as a free-tier customer.

Tutorials and Screencasts

• Illustrate regular, trouble-free usage of advanced topics
  • The Frontend Build Process: System Startup
  • Hot module replacement in a development environment
• (see this page for more basic video tutorials)

Extra Content, Design Tweaks

How to add static content

• How to add static HTML content to the mDIS dashboard with Vue

• The Vue.js Router software currently permits that you can put any static *.html files with extra content into the web/ directory and call them directly in your browser. This is demonstrated in the JavaScript tutorial REST API access.

• You can add HTML fragments to forms via the Vue Slot mechanism. See Vue Slot Tutorial, then run npm run build.

How to change the mDIS page design

For mDIS reports, the CSS code can be changed easily. See directory web/css/ and change files there.

For a single Vue page or Vue component, edit the <style> element in a .vue file, and run npm run build.

For all pages (or components), read the Vuetify theming guide first. It contains a lot of alternatives to change the design.

If you are adventurous, read this next. Open directory backend/node_modules/vuetify/src/stylus/components/ and change the layout by editing Vuetify's stylesheets ending in .styl. (These are Stylus stylesheets, not CSS stylesheets). Don't do this unless it is absolutely necessary. Test your modifications and then run npm run build.

Alternatively, try to edit the minified CSS files in web/css directly. Good luck with that.

Tooling

A new set of "Tooling" pages is under construction.

Applications and Extensions

Desktop Apps, Browser extensions, Command-line tools...

in no particular order

• Recommendations concerning the VS Code editor: Install some extensions: Vetur, Bracket Pair Colorizer, PHP CodeSniffer, Markdown Preview Enhanced... See this script on mDIS-installer: vscode-install-extensions.sh

• As a MySQL administration tool, we recommend: Adminer, (Installation). It can do 99% of basic DBA tasks. For more advanced tasks, such as renaming foreign keys, see phpMyAdmin or MySQL Workbench.

• For Vue2: The legacy Vue.js devtools Browser Extension - for visualizing the component structure on the mDIS data entry pages. See HowTo document.

• Postman: There existed an ICDP-internal "integration testsuite" for REST API calls, available as a Postman collection. (Older version .json file). It can automate create, inserts, updates, deletes, gets, logins. It also contains many tests and a test runner. This collection is available to all ICDP team members.

• NPM: npm run serve builds mDIS in a "development mode" which supports Hot Module Reloading and other developer productivity features. Global NPM Modules needed: @vue/cli, serve. See the package.json file in the GitLab repo.

Best practices, TBC

• If you go on a field trip, take offline documentation with you.
• Study mDIS thoroughly, because hacking the mDIS Codebase in the field requires self-confidence and deep understanding.

Browser Hacks

Sometimes it helps to "pin" a browser tab with the context menu. Select a tab, press (Shift + F10), click "pin".

Then the "Duplicate tab" feature might work just in some mDIS instances.

Contact Information

ICDP - ICDP Data management. Address is important to 'real' non-ICDP users.

For now, see https://gitlab.informationsgesellschaft.com/dis/dis/-/project_members

TBC: establish a real feedback channel for end-user communication.

References

ICDP Internal Wiki

• ICDP internal wiki:
  The Wiki hidden in the sidebar menu of mDIS repo at GFZ GitLab contains a lot of information and notes. In particular, very detailed instructions about mDIS installation workflows ("mDIS from scratch", "mDIS online to mDIS VBox", ...), but the wiki is not public at this time (2025).

Yii2 and PHP

• The Definitive Guide to Yii 2.0 Authors: Many, 2014-today. ("Living Document").
• Web Application Development with Yii2 and PHP. Authors: Mark Safronov and Jeffrey Winestett. Packt Publishers, 2014.

Web Programming, Sysadmin Technologies

• Frontend: Vue.js, Vuetify, JavaScript

• Frontend Build system: NPM, Webpack, Vuepress (Documentation generator)

• Backend: PHP-Stuff: Yii2, PHP, Composer, Packagist.org, Adminer, a free Database Management and Query Tool in a single PHP file.

• Backend: Shell Scripting: BashFAQ Wiki, Linux Doc Proj.,

• Virtualization: VirtualBox, Vagrant, Base Images used for mDIS:

  • currently: Ubuntu 24.04, Ubuntu 22.04
  • 2019-2022: ubuntu 18.04, ubuntu 20.04, on Vagrant

Identifiers

• IGSN consortium, IGSN Schema, How ICDP handles IGSNs, Geosamples.org at SESAR, ODM2 Controlled Vocabularies

Misc:

• PHP: Yii2 Popularity in 2020, Quick reference
• List of Emojis available for embedding into Markdown text (GitHub markdown has a different list, longer)
• Material Icons - Material icons are symbols for common actions and items.
• Twitter Bootstrap 3 - Governs look-and-feel for features that are rendered server-side, e.g. the mDIS User Administration page, some Single-Item Reports, and Labels.
• Pro Git (online), Authors: Scott Chacon and Ben Straub; Book published by Apress, 2014.
• "Awesome" Link Collections (plentiful but not always up-to-date): VueJS JavaScript Framework, Vuetify component library, Vuepress documentation system, PHP programming language, Composer PHP package manager, NPM Node Package Manager, Vagrant, Docker

---

continued in Scratch area