mDIS User Documentation

mDIS For Developers (ICDP internal)

mDIS For Developers

ICDP internal

This section is somewhat specific to ICDP internal mDIS-installations. However you might still find some pearls of wisdom here on this page.

Testsuites

In 2020, mDIS has grown in complexity, so automated software testing has become more important.

Note

This is also a system administration task.
You'll need shell access to run most of these tests, or access to the Continuous-Integration infrastructure, or be in physical posession of the POSTman suite (described in the next paragraph).

Of course you need valid mDIS credentials, too.

REST-API Tests

  • For testing the REST API of the PHP backend, there exist ICDP-internal testsuites, available as collections (opens new window) for the Postman (opens new window) test runner. The testsuites contain many tests of API calls provided by the mDIS PHP controllers. Postman testing automates the following actions: create items, insert, update, delete. login/logout. And, of course many different variants of GET operations.
    This collection is available to all ICDP team members.

Backend tests

  • The Yii developers recommend the end-to-end testing tool Codeception (opens new window). It is installed in mDIS' backend/vendor directory, because it is a third-party code dev-dependency of Yii2.
    To run the Codeception tests:
    • run all tests, create coverage reports, and show the results on the command line:
      ./backend/vendor/bin/codecept run --coverage-text --coverage-html --no-colors
    • run php backend/vendor/bin/codecept run functional or
    • run php backend/vendor/bin/codecept run unit.
      To enable the codecept testrunner, you must:
    • set up a test database dis_test (or similar) with some tables inside which match the real mDIS schema.
    • declare the following environment variables: MYSQL_PASSWORD, MYSQL_DATABASE, MYSQL_USER, MYSQL_HOST with appropriate values. Pro tip: create an .env file such that you can simply run . .env.

User-Interface (GUI) Tests

  • For testing the mDIS User Interface, ICDP Data Management has developed a few scripts for running GUI tests with Codeception and Selenium. However they are brittle. They tun only in a very specific configuration; and even then they run slowly. Ask us for access or assistance with these test scripts.

  • Optional. There are a few unit tests for Javascript code, currently for Datetime entry only. Run testsuites on the command line with

    • npm run test:unit or
    • npm run test:unit -- --watchAll
      (This is just a basic example without much test coverage.)

Fixing broken tests

If you have tests that fail, then either:

  • something is wrong with your mDIS installation, or:
  • the schema of the test database dis_test, which the Codeception test runner uses for a few generic tests, is different from the schema your database uses.
  • the mDIS codebase has evolved, but the tests have not been updated, or:
  • you have customized mDIS such that new forms, models, or tables have been created or updated. Thus, tests that assume a "vanilla" (= default) installation fail.

You must find out yourself what has happened.

To do that, run single tests, and/or run the tests with more verbosity.
To run single tests, perhaps do this:

php backend/vendor/bin/codecept run unit backend/tests/unit/templates/models/FilesUploadedFormTest.php

To run tests with more verbosity, add -vv or vvv to the command line.

php backend/vendor/bin/codecept run functional -vv

Debugging

Remote-Debugging PHP with XDebug 2 or 3

Debugging the mDIS Backend

Remote-Debugging means going through running code on a step-by-step basis.
Doing this with free PHP debugger "XDebug" is documented in PHP Debugging.

Debugging the mDIS REST API

The mDIS REST API is implemented in PHP, and is accessed via HTTP requests. Accessing The API is a backend task.

Debugging the mDIS VueJS Frontend

Use Node and the Chrome Developer tools

  • As a test run, npm run serve to start the development server. If the command fails, remove the node_modules/ directory and run npm install again.
  • change file vue.config.js at the following lines such that the devServer section ,matches your REST API endpoint:
module.exports = {
  //...

  // Webpack will proxy/forward all requests starting with /api to the mDIS backend.
  // this way you can debug the frontend on localhost, but the backend will run elsewhere.
  devServer: {
    proxy: {
      '/api': {
        target: 'https://wb31.gfz-potsdam.de/mdis/test23'
      }
    }
  },
}
1
2
3
4
5
6
7
8
9
10
11
12
13
  • On localhost, run npm run serve to start the development server.
  • Open Chrome and go to http://localhost:8080 (look at the output of npm run serve command).

    • Open the devtools (F12 or Ctrl+Shift+I)

    • Then open a new tab in Chrome and go to URL chrome://inspect.

    • Click "Devices" in the sidebar. In the main panel, Remote Target should be empty.

    • in a separate command line, run node --inspect. This will start the NodeJS debugger.

    • Chrome will detect this and put itself into "inspect" mode.

    • Remote Target: #LOCALHOST Target trace node[62291] file:/// inspect

    • click on "Open dedicated DevTools for Node" Icon in the Developer Tools.

Legacy Debugging (opens new window)
  • Or run node --inspect-brk node_modules/.bin/vue-cli-service serve to start the VueJS development server (built on top of Webpack).
  • Then click on "inspect" in the "Remote Target" section.
  • Then you can set breakpoints in the VueJS code.

(This has been deprecated with Node Version 7, and the protocol is no longer maintained, although it might still work.)

Official way of Debugging

Official Documentation (opens new window).
Proceed in a similar way as described above. Add the folder where you have stored the mdis code to the panel "Sources" in the Chrome Developer Tools.
Then you can set breakpoints in the VueJS code (the .vue files and .js files) and step through it.

TBC

Debugging VueJS - The VueJS DevTools

Install the Browser Extension "VueJS DevTools" from the Chrome Web Store.
These Tools do not enable "real" step-by-step debugging. Instead they offer a good way to get an overview of the VueJS component tree. Routes defined and used in the VueJS code are visible. Events and Event Handlers are also visible.

Visualizing the VueJS component tree

See this hint from our "VueJS DevTools" article.

The Yii2 Debug Bar

Intercepting SQL Statements processed by Yii

The Yii Debug Bar is a tool that intercepts SQL statements processed by Yii. YOu will see that a lot of permissions checks are done by Yii, and that the SQL statements are sent to the database server. For each query fetching science data there are 10 times as much permission checks. This is a lot of overhead but it is necessary to ensure that users only see data they are allowed to see.

Getting Diagnostic information about PHP <-> Database interaction

The Yii Debug-Bar is usually invisible. If it is enabled (by setting the environment variable YII_DEBUG = true) it will become visible on the mDIS Users-Manager page, as shown below in the screenshot.

It looks like this, and users must click to expand it from a minimized, collapsed state:

small

Expanded it looks like this:

bar expanded

Click on any field to expand the debug bar to an even larger debug-window. (Not shown here).

Then you can search for many things, e,g, stack traces, errors texts, and even SQL statements sent, by Yii, from the Webserver to the MySQL database.

To enable the debug bar, open a browser window with the official instructions (opens new window) and read it. Then follow along and do this:

Open file backend/config/web.php. Change this line of code

'bootstrap' => ['log', 'api', 'cg'],

to this:

'bootstrap' => ['log', 'api', 'cg', 'debug'],

This line of code is located at the top of the file, (in line 11 or so).

Add this block of code to the modules config section in web.php:

        'debug' => [
           'class' => 'yii\debug\Module',
           'allowedIPs' => [env('EXTERNAL_ALLOWED_IP'), '127.0.0.1', '::1']
        ]
1
2
3
4
5

Set the following variables in file (project_root)/.env, or set them manually inside web/index.php:

    YII_DEBUG=true
    YII_ENV=dev
    EXTERNAL_ALLOWED_IP=139.17.*.*
1
2
3

Troubleshooting the debug bar

If you still cannot see the Yii debug bar, then you should check if Yii has started collecting the debugging info anyway. Maybe Yii just cannot display the Debug Bar GUI Widget for some reason.
Check the backend/runtime/debug/ directory and have a look at the newest *.data files. These are text files in PHP's serialization format. They are not convenient to read, but you still can use them to find out what is happening. The grep tool can go a long way with this.

See also "Logfiles" below.

Optional (?):

As an alternative to the debug bar, install (opens new window) Apache's mod_security2 module. This allows you to trace the payload of PUT and POST requests, as they arrive in the container.

Do not enable the Debug-Bar feature in production. It is probably bad for performance. In mDIS you need to have administrator privileges to see it (and understand it).

Remove older sess_* files from the directory specified in PHP's session.save_path, e.g. /var/lib/php/sessions. (I don't remember when and why this helped me once).

SQL Debugging on mysql

A simple way to find long-running SQL queries is to use the query SELECT * FROM sys.statement_analysis LIMIT 10, or work directly with the tables in database performance_schema. You need mysql 8 (or equivalent) for this, and your user needs access to the performance_schema and sys databases.


If you are using a Mysql variant as a backend database for mDIS, then you can also use the CLI tool `mysqldumpslow` to analyze all slow SQL statements issued to the database server.
The `mysqldumpslow` command line client is part of the `mysql` package. It is installed by default, but it is not enabled by default.

```sql
-- SQL: Verify if Slow Query Log has been configured
show global variables like '%slow%';
1
2
3
4
5
6
7

You must enable it by adding a slow_query_log line in the my.cnf file. Then you must restart the MySQL server. Then you can use the mysqldumpslow command line client to analyze the slow SQL statements. You will get the sql text of slow-running queries, and you will get a summary of the slow queries, sorted by the number of times they were executed, or by the total time they took, and which user issued them.
You should try this first on a local development machine, especially if you have more than the mDIS database on your local MySQL server. You might easily write much more text to the logfile than intended.
A more detailed explanation is out of the scope of this article. /But you can find some instructions (opens new window) on the Internet. / //To be continued...//

Docker: Tips and Tricks

Getting into an mDIS Docker Container

wb45 only: Enter the running Docker container (the server-side, the backend) with:

docker-volume up -d                # start all containers
docker exec -it disapp_php_1 bash  # run bash inside webserver container
1
2

Logfiles

Logfiles in the Docker Container

On the docker host, display the Apache Error Log with

docker logs -f <containername>

Although Apache Log files are in the container's /var/log/apache2/ directory, they are symlinked to stderr, and therefore are not displayable in the container.

Logfiles in the mDIS instance

Viewing mDIS webserver- and PHP logfiles on Linux (inside a Virtualbox instance or inside a Docker container):

     cd /var/www/dis/backend/runtime/logs
     tail app.log  # mdis yellow toast-boxes, error log
     tail convert.log # imagemagick writes to this logfile.
     #grc tail convert.log # grc: logfile colorizer
1
2
3
4

Accessible from outside Docker container, too, when /var/www/dis is mounted as a Docker volume.

Docs

Offline reading

Offline reading of a 2020 version of this documentation is very easy by downloading the PDF version (opens new window). This contains everything as a single PDF file, but the content of the PDF might be slightly older than these web pages you are just reading.

Advanced: git-clone the Gitlab repository (opens new window), and read *.md files opened from your local directory. These are text files written in Markdown. The .md files are human readable.

Optional, for developers: To generate html files locally on your computer, install Vuepress (opens new window) with npm i -g vuepress, and inside the repository's directory, run vuepress build. Then perhaps run one of the following commands to open the documentation as a web page in your browser:

  • Python 2: python -m SimpleHTTPServer 8000; firefox localhost:8000

  • PHP: php -S localhost:8000; firefox localhost:8000 Doc Page (opens new window)

  • Javascript/NodeJS: npx lite-server # by default listens on port 3000 These commands start small web servers that are built into the respective language interpreters (Python/PHP), or downloads a small webserver lite-server (opens new window) from the internet (NodeJS). You might need to install lite-server with npm i -g lite-server (global) or npm i -g lite-server --save-dev (local).

    Big list of http static server one-liners (opens new window)

Searching this Documentation

See the text field with the looking-glass icon [🔍], located in the title bar at the top of this webpage.

Searching with Vuepress

There is a search engine that comes built-in with Vuepress, but it is not very powerful. It only searches headlines and keywords that appear in the sidebar.

We have replaced vuepress search with an index provided by the commercial indexing service Algolia. They have a free tier that we are using.

Searching with Algolia.com

Algolia-based search supports fulltext search, similarity search, and it has a nice user pull-down interface.

The documentation site must be indexed with their scraper and pushed to algolia.com.

The indexing script:

#!/bin/sh
cd /home/knb/code/git/dis-docs/_work/dis-search/
# on wb45, the actual command to run the algolia indexer.
# details are in the .env file and  prod_mdis.json file
docker run -it --env-file=.env -e "CONFIG=$(cat /home/knb/code/git/dis-docs/_work/dis-search/config/prod_mdis.json | jq -r tostring)" algolia/docsearch-scraper
cd -
1
2
3
4
5
6
7

Config file /home/knb/code/git/dis-docs/_work/dis-search/config/prod_mdis.json for the Algolia scraper + indexer:

{
  "index_name": "prod_mdis",
  "start_urls": ["https://data.icdp-online.org/mdis-docs/guide/"],
  "selectors": {
    "lvl0": "#app main.page h2",
    "lvl1": "#app main.page h2",
    "lvl2": "#app main.page h3",
    "lvl3": "#app main.page h4",
    "lvl4": "#app main.page table",
    "text": "#app main.page p,#app main.page ul,#app main.page ol, #app main.page td"
  }
}
1
2
3
4
5
6
7
8
9
10
11
12

TIP

If the search query yields a 404 error, remove the duplicated section from the URL search result page. Often there is a duplicated path-fragment in there. Change the URL fragment from mdis-docs/mdis-docs/ to mdis-docs/ and reload the 404 page.

Run the indexing script again. It must be run from within directory dis-docs/_work/dis-search/ to create correct URLs.

The .env file mentioned in the script contains an Algolia APP-ID and and API key to give access as a free-tier customer.

Tutorials and Screencasts

  • Illustrate regular, trouble-free usage of advanced topics
    • The Frontend Build Process: System Startup
    • Hot module replacement in a development environment
  • (see this page for more basic Video tutorials)

Extra Content, Design Tweaks

How to add static content

How to change the mDIS page design

For mDIS reports, the CSS code can be changed easily. See directory web/css/ and change files there.

For a single Vue page or Vue component, edit the <style> element in a .vue file, and run npm run build.

For all pages (or components), read the Vuetify theming guide (opens new window) first. It contains a lot of alternatives to change the design.

If you are adventurous, read this next. Open directory backend/node_modules/vuetify/src/stylus/components/ and change the layout by editing Vuetify's Stylesheets ending in .styl. (These are Stylus (opens new window) Stylesheets, not CSS Stylesheets). Don't do this unless it is absolutely necessary. Test your modifications and then run npm run build.

Alternatively, try to edit the minified css files in web/css directly. Good luck with that.

Tooling

A new set of "Tooling" pages is under construction. PHP Debugging

Applications and Extensions

Desktop Apps, Browser extensions, Commandline tools...

in no particular order

  • Recommendations concerning VS Code editor: Install some extensions: Vetur, Bracket Pair Colorizer, PHP CodeSniffer, Markdown Preview Enhanced... See this script on mDIS-installer: vscode-install-extensions.sh (opens new window)

  • As Mysql administration tool, we recommend: Adminer (opens new window), (Installation (opens new window)),. It can do 99% of basic DBA tasks. For more advanced tasks, such as renaming foreign keys, see phpMyAdmin (opens new window) or Mysql Workbench (opens new window).

  • The VueJS devtools Browser Extension - for visualising the component structure on the mDIS data entry pages. See HowTo document.

  • Postman: There exists an ICDP-internal "integration testsuite" for REST API calls [, available as a Postman (opens new window) collection. (Older version .json file). It can automate create, inserts, updates, deletes, gets, logins. It also contains many tests and a test runner. This collection is available to all ICDP team members.

  • NPM: npm run serve builds mDIS in a "development mode" which supports Hot Module Reloading, and other developer productivity features. Global NPM Modules needed: @vue/cli, serve. See package.json file in Gitlab repo.

Best practices, TBC

  • If you go on a field trip, take offline documentation with you
  • Study mDIS thoroughly, because adapting it in the field requires self-confidence and deep understanding

Browser Hacks

Sometimes it helps to "pin" a browser tab with the contextmenu. Select a tab, press (Shift + F10), click "pin".

Then the "duplicate tab" feature might work just in some mDIS instances.

Contact Information

ICDP (opens new window) - Address is important to 'real' non-ICDP users

For now, see https://gitlab.informationsgesellschaft.com/dis/dis/-/project_members (opens new window)

TBC: establish a real feedback channel for end-user communication

References

Books and Wikis

Websites

Misc:

continued in Scratch area