mDIS User Documentation

mDIS inside Docker - Part 2

Summary

We will adapt a pre-built Docker image (opens new window) . Add some custom software components. Customize the mDIS code, and even patch some 3rd party dependencies as necessary.

The final result will be a new mDIS instance containing the basic mDIS features including user login/logout, a Dashboard, a data entry system, among other features.

mDIS inside Docker - Part 1 of this guide.
Native Installation on Linux
Installation on Virtualbox: general, detailed
Shellscripts in the mdis-installer (opens new window) gitlab repository

Purpose

  • How to get mDIS Code from Docker and Gitlab
  • How to build a brand new mDIS v3 instance from scratch.
  • Add a new container to a Docker network behind a reverse proxy.

Final result: Multiple Docker Containers behind a single Apache Reverse Proxy.

See this figure:
mDIS Apache as Reverse Proxy
We will add a new box to the middle row, and also tell Apache and the MariaDB about it.

Requirements

For installing mDIS, the minimum requirement was that your Web server supports PHP 7.1, released in 2016. Current minimum PHP for which yii2 Docker Images are available is PHP 7.4. (released in 2019, supported until Nov 2022).

This walktrough uses a PHP 8 version.

WARNING

PHP 8.0 is fully supported, PHP versions 8.1 and higher need some tweaking of the mDIS source code.

Installation

Install Docker and Docker-Compose. Download the latest docker-compose software from https://docs.docker.com/compose/install/.

Install with Docker - ICDP internal

Step-By-Step Guide, ICDP Internal

Install with Docker

Very verbose instructions. For the impatient, see Step-By-Step Guide; and for other platforms, check the mdis-installer (opens new window) repository.

Learning Docker

Learn as much as you can about containerization and the Docker ecosystem (except Docker Swarm, perhaps).

Optional: Register your Docker-Hub account at https://hub.docker.com/u/<your_username>. This makes it easier to pull images from Docker Hub.

Download an appropriate yii2 php-apache image from Docker Hub (opens new window).

This version uses the yii2-php/8.1-apache image. but you are free to try any other image (e.g. the PHP 7.4 version used to work for many years). See below for PHP 8.1 issues, and how to fix them.

You can also tweak the image by adding the following to your Dockerfile.
Alternatively, add some extra software with commands from within the running Docker container. Then execute docker commit commands to create your own customized Docker image, (which will still be based on the yii2-php/8.1-apache image).

The following sketches some commands that add software to the newly downloaded image.

TBC

adduser ... # with disabled password
# add user to sudoers group

# install nvm
# install node 16 --lts

# install binaries mDIS needs, e.g. imagemagick,
# but also some of your own preferred developer tools
# that should be globally available, 
# e.g. diff-so-fancy and other tools

# set umask
# set bashrc and aliases
# ...
1
2
3
4
5
6
7
8
9
10
11
12
13
14

mDIS Installation from Git Repository

To proceed, you must clone the mDIS repository. However, do not install mDIS itself yet. Start with installing the required 3rd-party software. What you need is listed in the composer.json file.

Clone the mDIS repository and change into the newly created subdirectory (most likely dis).

Optional: Set environment variables for the container

TBC

Environment variables

We use environment variables to pass mySQL Credentials into the running container. These env vars are MYSQL_USER, MYSQL_PASSWORD, MYSQL_HOST, MYSQL_DATABASE.

  • set them outside of the container in .env files
  • use .env-files inside docker-compose.yml files
    • or set them in your .bashrc
    • or set them in /etc/environment
    • or set them in /etc/apache2/envvars
    • pass them along with docker run/exec commands ...

You can also hardcode credentials in your Dockerfile, or set them inside the container when it is running. Do what works best for you. For long-term consistency and reproducibility, you can use .env files. You can store these externally at a safe, central location. This way you can lookup forgotten credentials and passwords later.

TBC

## TODO: add some commands clarifying .env file use
1

Example of a Docker-Compose file

version: "2"
services:
  php_generic22:
    env_file:
      - ./.env
      - ./mdis-generic22/.env-mysql
    
    image: yii2-php/8.1-apache
    # alternatively use a customized docker image, 
    # built on top of the yii2-php/8.1-apache image:
    # image: localhost:5000/mdis-v3-blank:20220406
    
    volumes:
      - generic22_web:/var/www/dis
      - ~/.composer-docker/cache:/root/.composer/cache:delegated
      - ./:/app:delegated
    ports:
      - "8052:80"
      
    command: "apachectl -D FOREGROUND"
    # if apache is not running or not installed:
    #command: "tail -f /dev/null" 
    restart: unless-stopped

volumes:
    generic22_web:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

Prepare .env file in advance

Prepare docker environment file by copying .env-dev to .env. Docker will pick up any values from the .env file mentioned on line 5 of in the Docker-Compose file. You can put environment variables in the .env file. e.g. MYSQL_USER, MYSQL_PASSWORD, MYSQL_HOST, MYSQL_DATABASE.

Update your vendor packages

Run the installation triggers (creating cookie validation code)
docker-compose run --rm php composer update --prefer-dist
docker-compose run --rm php composer install

Newer method

Using docker-compose with custom file from snippet above docker-compose -f docker-compose.dive22.yml --project-name mdis up -d --no-recreate

This will start up a new container mdis_php_dive22_1. It will be based on a preconfigured Docker image (which you must provide) as the basis of the installation.

In the docker-compose.*.yml file, I personally like to put credentials into two .env* Files, like this:

    env_file:
      - ./.env                    # generic ENV Variables for all containers
      - ./mdis-dive22/.env-mysql  # MYSQL_* ENV Variables for the dive22 container
1
2
3

If installation fails, do this:

(TBC)

Problems with PHP 8 and PHP 8.1

The following workaround makes the installation of the 3rd party PHP dependencies possible and allows to login and to work with mDIS forms.
We have not (yet) checked if mDIS actually works on PHP 8.1 in the long term, in production. It seems to work for now, during internal use of PHP8.1-based mDIS instances, and during mDIS instance preparation.

  1. Fork the "ancor/yii2-related-kv-storage" (opens new window) Repository, containing an (apparently abandoned) Composer Package.
  2. cd into a suitable subdirectory, e.g. icdp_work/packages/.
  3. git clone the forked repository.
  4. cd into subdirectory yii2-related-kv-storage.
    My subdir is <DIS_DIR>/icdp_work/packages/yii2-related-kv-storage
  5. change one line (line 14) in composer.json
    from: "php" : "^7.0",
    to : "php" : "^8.0", # pump to higher major version
  6. (optional:) git push the repo back to your own GitHub repo
  7. cd into the dis directory (/var/www/dis)
  8. edit /var/www/dis/composer.json. Change this:

from:

    "repositories": [
        {
            "type": "composer",
            "url": "https://asset-packagist.org"
        }
    ],
1
2
3
4
5
6

to:

    "repositories": [
        {
            "type": "composer",
            "url": "https://asset-packagist.org"
        },
        { "type": "path", "url": "icdp_work/packages/*" }
    ],
1
2
3
4
5
6
7

Run composer update again. This will now install ancor/yii2-related-kv-storage by also looking into your local subdirectory, and installing the "patched" version.

Run post project creation triggers (set required permissions).

From outside the container:
docker-compose run --rm php composer run-script post-create-project-cmd
From inside the container:
composer run-script post-create-project-cmd

Add missing return-type Annotations to 3rd party code

PHP 8.1 might not be able to infer the correct return type of some functions in file backend/vendor/ancor/yii2-related-kv-storage/src/Config.php. Then PHP 8.1 will throw exceptions and mDIS will crash.

Change these function declarations

From

    function rewind() 
    function current()
    function key()
    function next() 
    function valid() 
    public function count()
    public function offsetSet($offset, $value)
    public function offsetExists($offset)
    public function offsetUnset($offset)
    public function offsetGet($offset)
1
2
3
4
5
6
7
8
9
10

to these:

    function rewind():void 
    function current():mixed 
    function key():mixed 
    function next():void 
    function valid():bool 
    public function count():int
    public function offsetSet($offset, $value):void
    public function offsetExists($offset):bool
    public function offsetUnset($offset):void
    public function offsetGet($offset):bool
1
2
3
4
5
6
7
8
9
10

Otherwise some PHP-8 versions might throw deprecation notices as Fatal Errors. TBC

Install Frontend Dependencies

To build the mDIS frontend, quite a lot of knowledge is required about the JavaScript ecosystem.

Javascript code

npm install

If you happen to run the mDIS Build-System with Node 16 (stable release in 2022), the above-mentioned command will almost succeed, but will fail after having done 90% of the work.
There is a workaround, however. You need to run a custom script:

node install-node-sass_sass-loader.js

and the installation process will continue. If it doesn't, try something else.

Finally, run npm install again until you see output similar to this:

added 1911 packages, and audited 1912 packages in 13s

43 vulnerabilities (1 low, 20 moderate, 17 high, 5 critical)

Run `npm audit` for details.
1
2
3
4
5

Generate frontend bundle

To generate frontend bundle, use one of the following commands

npm run build # produces a production-ready bundle in the /web directory (see https://cli.vuejs.org/guide/cli-service.html#vue-cli-service-build)
## Alternative command, for advanced users and hardcore developers:
npm run serve # starts a dev server (see https://cli.vuejs.org/guide/cli-service.html#vue-cli-service-serve)
1
2
3

Start the containers

# add -d to run as daemon (in the background)
docker-compose up -d --no recreate
1
2

You can then access the application at http://127.0.0.1:<port-num-from-docker-compose-file>

After adding nodejs to the container, first build time was increased. Take the build option out and uncomment the image line if do not need nodejs in the container. check here (opens new window) and here (opens new window) for suggested solutions.

On the command line you should see something like this:

docker ps | grep dis

Result:

CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                    NAMES
71b25a7418dd        yiisoftware/yii2-php:7.1-apache   "docker-php-entrypoi…"   23 hours ago        Up 13 seconds       0.0.0.0:8000->80/tcp     dis_php_1
57cdf5ca735e        mariadb                           "docker-entrypoint.s…"   23 hours ago        Up 23 hours         0.0.0.0:8001->3306/tcp   dis_db_1
7424a0f9f6a7        adminer                           "entrypoint.sh docke…"   3 days ago          Up 3 days           0.0.0.0:8002->8080/tcp   dis_adminer_1
1
2
3
4

Rightmost Column NAMES signifies:

  • dis_php_1 - Webserver with PHP/Yii2 Application
  • dis_db_1 - Mariadb, Database Server
  • dis_adminer_1 - Adminer, Web-based Database Administration Tool

Optionally, stop the containers with docker stop dis_php_1 dis_db_1 dis_adminer_1.

PHP 8.1

PHP 8.1 performs stricter compile-time checks and than PHP 8.0 and PHP 7.4.

In ./yii and ./web/index.php, change this line from

$errorReporting = error_reporting(E_ALL);
1

to

$errorReporting = error_reporting(E_ALL  & ~E_DEPRECATED & ~E_STRICT);
1

Optional: Near Line 497 of vendor/yiisoft/yii2-gii/src/Generator.php

Add a single line of code by changing this snippet from

 public function generateString($string = '', $placeholders = [])
    {
        $string = addslashes($string);
1
2
3

to

 public function generateString($string = '', $placeholders = [])
    {
        if(is_null($string)){ $string = '';}
        $string = addslashes($string);
1
2
3
4

Patch some PHP files

This bug might already be fixed when you read this.

In directory /var/www/dis/backend/modules/cg/generators/DISModel/specializations/, remove or uncomment calls to parent::...().
PHP 8 compile-time checks fail if this is not done in these files:

Line 27 of BaseCurationCuttings.php Line 42 of BaseCurationSampleRequest.php Lines 17 and 224 of BaseArchiveFile.php

Database migrations

Apply database migrations to create required tables, and "seed" (fill/insert) those tables with data.

# bash into container
docker exec -it dis_php_1 bash
# then call yii migrations
yii migrate
# seed desktop widgets
yii seed/widgets
# seed users accounts
yii seed/users
# optional: load DSEIS data - really old database dump
yii seed/example-dump 
# optional: forms permissions
yii seed/form-permissions
1
2
3
4
5
6
7
8
9
10
11
12
13

TODO: list and document all available "yii seed" commands

Directory permissions

Quite a few directories must be made world-writablke, or writable for the www-data user.

TODO: Document which diretories should have a certain ownership and specific write permissions.

Docker Volumes

The mariadb container stores persistent DIS data inside volume dis_dbvolume:

docker volume inspect dis_dbvolume

Docker Volumes enable the container to "survive" restarts. The volume persists data created by both the yii migrate command and data eventually entered by the DIS user.

Notes:

  • Minimum required Docker engine version: 17.04 for development (see Performance tuning for volume mounts (opens new window)). Most extensively use were Docker-Engines v19 and v20.
  • The default configuration uses a host-volume in your home directory .composer-docker for-composer caches

Configuration

Database

This configuration may not always be necessary If you have a Docker-Compose based installation, which sets environment variables via .env, and sets up linked containers,the following might not be configured.

Edit the file config/db.php with real data, for example:

return [
    'class' => 'yii\db\Connection',
    'dsn' => 'mysql:host=your_host;dbname=your_db_name',
    'username' => 'your_db_user',
    'password' => 'your_db_password',
    'charset' => 'utf8',
];
1
2
3
4
5
6
7

Upgrading an existing mDIS v2 to mDIS v3

Background

In January 2022 we decided to remove several files from the repository that where automatically generated or usually modified in an instance. Additionally several separately developed branches have been merged including the new standard data model of ICDP. The Branch "mDIS V2" contains the version before those breaking changes.

Steps

To update an existing instance to the new Version 3.x, please follow these steps:

  1. Make sure you are on master branch, type git pull in order to update the project.

    • If you do a git pull, various modified files will probably be criticized.
      • All files in backend/models/base can be restored via "git restore" (they will be deleted on pull)
      • All files in backend/models and backend/forms that have not been specialized can be reset (they will be deleted on pull)
      • All files src/forms/*.vue.generated can be reset (they will be deleted on pull)
      • The specialized files in backend/models/ and /backend/forms/ can be detached from the repository: git rm -cached <path-to-file-that-dont-want-to track>
    • Now git pull should work

  2. Type composer install in the console.

  3. Type ./yii upgrade 3 in the console.

    • This command will do the following:
      • Copy the default models templates to ‘backend/dis_templates/models’.
      • Copy the default forms templates to ‘backend/dis_templates/models’.
      • Create the missing tables in the data base according to the already copied models templates.
      • Apply the migrations of ‘usuario’ library.
      • Update the Php models and form as also the vue.generated files.

  4. Delete the folder node_modules from your project root.

  5. Type npm install in the console.

  6. Type npm run build in order to create a new build or npm run serve to serve.

Apache Configuration

Running Apache as a Reverse Proxy

Work in Progress

Configuration

Only a rough guide

Do not copy these settings literally. They are only a template. You must adapt the exact numeric values of port numbers, and the path-fragments, to your needs.

This change in /etc/apache2/apache2.conf might be needed to avoid weird log messages in some versions of Apache.

# apache.conf
# The accept serialization lock file must be stored on a local disk (?)
#
Mutex file:${APACHE_LOCK_DIR} default
1
2
3
4

For each mDIS instance, insert these stanzas into sites-available/my-site.conf

    SSLEngine On
    SSLProtocol +TLSV1.2 +TLSv1.3
    ProxyPreserveHost On
    ProxyRequests Off
    SSLProxyEngine On

    # This is a typical setup for an mDIS instance running behind a reverse proxy.
    RedirectMatch permanent ^/mdis/tadp /mdis/tadp/                             
    ProxyPass "/mdis/tadp/" http://0.0.0.0:8056/                                
    <Location /mdis/tadp/>                                                      
        ProxyPassReverse /                                                      
        ProxyHTMLEnable On                                                      
        RequestHeader unset Accept-Encoding                                     
        ProxyHTMLURLMap  http://0.0.0.0:8056/  /mdis/tadp/                      
        ProxyHTMLURLMap / /mdis/tadp/                                           
        AddOutputFilterByType SUBSTITUTE application/json 

        # Sadly, we need the following line for in-place edit of some json-outputs.
        # This requires mod_substitute to be installed.
        # See https://httpd.apache.org/docs/2.4/mod/mod_substitute.html
        Substitute "s|http://data.icdp-online.org/|https://data.icdp-online.org/mdis/tadp/|n"
    </Location>

    # If you use the SQL Admin tool in its own container,
    #  you can use the following lines to redirect all SQL requests 
    # to the SQL Admin tool.
    RedirectMatch "/adminer.php$" "/adminer"

    # not an error, not redundant
    ProxyPassMatch /adminer/adminer.css.* http://0.0.0.0:8004
    ProxyPassReverse /adminer/adminer.css http://0.0.0.0:8004

    # match with session key qs-param
    ProxyPassMatch /adminer.css.* http://0.0.0.0:8004
    ProxyPassReverse /adminer.css http://0.0.0.0:8004

    ProxyPassMatch /adminer$ http://0.0.0.0:8004
    ProxyPassReverse /adminer http://0.0.0.0:8004
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

Optional

Useful for Apache Monitoring tools

Optional: If you want to enable the Apache Modules status and info, you need to add these two lines to file web/.htaccess;

RewriteCond %{REQUEST_URI} !=/server-status
RewriteCond %{REQUEST_URI} !=/server-info
1
2

before the other lines starting with RewriteCond.

PHP Deprecation Warnings

Another potential problem:

These errors happen when error_reporting includes E_DEPRECATED (which is the default since PHP 8.0):

PHP Deprecated:  Return type of
 Ramsey\Uuid\Uuid::jsonSerialize()
 should either be compatible with 
 JsonSerializable::jsonSerialize(): mixed, 
 or the #[\ReturnTypeWillChange] attribute should be used to 
 temporarily suppress the notice in /home/ilovemistakes/work/uuid/src/Uuid.php on line 216

PHPUnit 8.5.23 by Sebastian Bergmann and contributors.
1
2
3
4
5
6
7
8

About this specific JSON-serialization deprecation warning see here (opens new window):

With the next update of PHP (8.1) the JsonSerialize::jsonSerialize() method will now have this signature: public function jsonSerialize(): mixed; )

This can lead to DEPRECATED: warnings appearing in the HTML output. And this can lead to "Headers already sent" errors and nasty-looking error pages, with big stack traces.

Workaround: set error_handling in php.ini to

error_reporting=22527
; same as
; error_reporting=E_ALL  & ~E_DEPRECATED & ~E_STRICT
1
2
3

This also needs to be set in the yii and yii.bat command line tools, and in web/index.php. All files containe lines that must be changed from

from

# from:
$errorReporting = error_reporting(E_ALL);
# to: 
$errorReporting = error_reporting(E_ALL  & ~E_DEPRECATED & ~E_STRICT);`
1
2
3
4

or similar. (I'm not so sure about the E_STRICT)

Tweaking PHP

Optional.

You can add values similar to these to your php-ini configuration file. They increase some resource limits that users might encounter while uploading files, displaying multi-page reports, etc.

# these improve memory usage and webserver performance
upload_max_filesize = 32M 
post_max_size = 48M 
memory_limit = 256M 
max_execution_time = 600 
max_input_vars = 3000 
max_input_time = 1000

error_reporting=22527
1
2
3
4
5
6
7
8
9

Some Linux distributions have these values in a dir /usr/local/etc/php/conf.d/, e.g. as y80-icdp-tweak.ini. but you might as well put them into /etc/php/conf.d/path/to/inifiles.

Adapt mySQL database

Optional: Copy users from old version

Copy the contents of the tables users, profile, and auth_assignment from the old version of the database to the new version. Copy them in this order (optional).

Test if you can login to the mDIS login screen, with your credentials.

Do this only if you want to work with your existing user accounts.

Run extra yii Migrations:

Optional

./yii seed/widgets - put the widgets on the dashboard

Import list values into mDIS

As of 04/2022 a working migration script for this task does not exist.

You have to insert the values manually. SQL Import works best. Fill the tables dis_list and dis_list_item, in this order.

You can also use the ListValuesImporter in the Upload Files menu in the mDIS Sidebar. This will allow you to import CSV Files into dis_list_item.

If you wanted to create a new list first (en entry in dis_list), you must do this manually via an SQL Administration tool. Remember the newly assigned id. Use that value as a list_id column for your CSV file-import file (for the dis_list_item table).

TBC

Troubleshooting Docker-Containers

Your mDIS Installation should be complete. Read the following only if you are stuck.

Check general configuration

Check settings in file

/etc/docker/daemon.json

Advanced stuff TBC

After rebooting the host server, not all containers might come up correctly.

Troubleshooting DB server / container

Check the new IP Address of the mysql-db container. Check the logfiles of your tools ( Nagios or similar). Adapt the Nagios monitoring command if necessary.

Troubleshooting container with HTML frontend for Docker registry

Enter the container, start Apache manually

service apache start or service apache reload or

Troubleshooting other servers

All other containers can be checked with their friendly container name, e.g. mdis_php_grind_1:

On wb45 run:

sudo -u Nagios /etc/nagios3/conf.d/check_docker_by_ssh -c mdis_php_grind_1

If this doesn't work, then run on rz-vm412:

$HOME/bin/docker_nagios/dkc_status.sh -c mdis_php_grind_1

Expected output:

OK: mdis_php_grind_1 status is running

Based on script (produces slightly more output): $HOME/bin/docker_nagios/check_docker --connection /var/run/docker.sock --cpu 10:20 --timeout 4 2>/dev/null

Connect to Docker Daemon via encrypted TCP socket

Communications between Docker Host and Docker containers are insecure, by default. It is recommended to encrypt the connections. Certificates and 3rd-Part Certificate Chain files are in

TBA 😃

Check if TCP socket connection is enabled for Docker daemon.

Port 2376 must be open. This port is the standard port for an encryptyed TCP socket connection. For security reasons, it should only accept connections from localhost or from hosts that you trust.

Command sudo ufw status numbered should return a firewall rule that allows accessing this port from the Nagios host.

[7] 2376 ALLOW IN 139.17.<...>

Check if port is open:

# as root on localhost
nmap --open localhost

# or more specifically
sudo nmap -PN -sT -p 2376 139.17.229.12

PORT     STATE SERVICE
2376/tcp open  Docker
1
2
3
4
5
6
7
8
9

TLS/SSL

For encrypted communications, the certificates and keys must be stored in a directory. Alternatives are:

/etc/ssl/certs/       # Debian/Ubuntu standard dir
/usr/share/ca-certificates/       # for your own certiicates
/usr/local/share/ca-certificates/ # and chain files
1
2
3

TBC