mDIS User Documentation

ICDP Internal: Installation of a new mDIS instance using Docker

Introduction

As of early 2023, this guide describes how to install a new mDIS instance at ICDP, manually. The installation procedure uses the Docker containerization technology. The installation is done on the ICDP server, and the new instance is accessible from the public internet. This is the most controlled but also most time-consuming way to install mDIS.

Summary

This guide below describes the manual installation procedure. An automated procedure (opens new window) is described in the mDIS-repo wiki (opens new window) on gitlab.gfz-potsdam.de.
Also see "mdis-blank" Repo (opens new window).
Gitlab's built-in Continuous Integration and Continuous Deployment (CI/CD) features enable this automated installation process.

Steps for manual mDIS installation from the source code

  1. (optional) If you prefer to work locally: Mount remote directory:
    sshfs rz-vm412.gfz-potsdam.de:/data1/docker/mdis-images /mnt/rz-vm412/mdis-images/
  2. cd to the mdis-images directory, or open it in a file manager.
  3. Copy and paste an appropriate (recent?) existing docker-compose-YYY.yml file from the ICDP directory
  4. Rename the file to docker-compose-XXX.yml where XXX is the name of the new instance. If your drlling project is named Plutonia then, e.g, plutonia would be a good value for XXX.
  5. The Image: instruction in the docker-compose-XXX.yml file could refer to an existing Docker image, e.g. git.gfz-potsdam.de:5000/mdis-hub/mdis-php-foundation:latest. This is a Debian 11 image, based on the official Docker image yiisoftware/yii2-php:8.1-apache. It contains most of the required and optional PHP8.1 3rd party packages and a selection of Linux packages needed for mDIS v3, but no mDIS 3rd party dependencies and no mDIS code. You can also try an mDIS-blank image.
  6. Search and replace all occurrences of YYY with XXX in the file.
  7. If you are using a reverse proxy: Assign a new port number to the XXX instance in the docker-compose-XXX.yml file. This is the port number that the reverse proxy uses to access the new instance.
  8. (optional) On your dev machine use sshfs, ssh rz-vm412.gfz-potsdam.de; cd /etc/apache2/sites-available
  9. If you are using a reverse proxy: Add a new stanza to the Apache config file. The stanza looks like this:
        ProxyPass /XXX https://data.icdp-online.org:Portnumber
        <Directory>....</Directory>

        (except that the stanza is longer, the port number is newly assigned.)
1
2
3
4
  1. Restart Apache
  2. Copy or create an .env-mysql file: The file is in the ICDP directory. The file name could be .env-mysql. Add this name to the docker-compose.XXX.yml file in section services/ php_fardeep23:/env_file:. Change the MYSQL_DATABASE= name to the database of the new instance. The file contains the MySQL credentials for the new instance. The credentials will be used by the php/apache process inside the Docker container, to access the MySQL database. You can reuse a file from an existing instance: cp mdis-test23/.env-mysql mdis-demo23/, but you should change the mysql credentials.
  3. (optional) You can also add Gitlab credentials here (or in a different .env file), e.g as PHP_ENABLE_XDEBUG= such that they become environment variables inside the Docker container. (A new empty database will be created later in step ~60).
  4. On data.icdp, create an exchange directory for the new instance:
    mkdir /data1/docker/mdis-images/XXX.
  5. Run docker-compose -f docker-compose.XXX.yml --project-name mdis up -d --no-recreate.
    This initializes the new instance. Ignore any warnings about "orphan containers". A new instance should be running, but it is not functional yet.
  6. (optional) In the .bash_aliases file on the server, add two new aliases to make entering the new instance easier. The aliases should look like this:
        alias  enter-XXX='docker exec -u mdis -it XXX bash'
        alias enter-XXXr='docker exec -u root -it XXX bash'
1
2
  1. Run source .bash_aliases to make the aliases available.
    1. Run enter-XXXr to enter the new instance as root.
  3. (optional) As root, run npm install -g diff-so-fancy (ICDP internal).
  4. (optional) Update npm: npm install -g npm.
  5. Inside the new instance, pwd should return /app; run ls /app to check that the exchange directory with the host has been mounted as a Docker volume, and many *.yml files and mdis-* subdirectories are there.
  6. Inside the new instance, ls /var/www should return dis html, and dis should be empty.
  7. chown www-data:mdis /var/www/dis; chmod 2775 /var/www/dis.
  8. (optional) Check that umask is set to 002 in /etc/bash.bashrc. Set it if it is not.
  9. exit the container as root
  10. enter as mdis: enter-XXX
  11. check that MYSQL_* environment variables have been set by Docker, from the .env-mysql file: env | grep MYSQL_
  12. (optional) customize the bash prompt: vi ~/.bash_it/themes/bobby/bobby.theme.bash; change \h to something of your liking, e.g. demo23. exit/logou the container as mdis. Re-enter as mdis to see the new prompt (and make sure the prompt is displayed properly).
  13. Run cd /var/www; git clone https://$GITLABUSERNAME:$GITLABPASSWORD@gitlab.informationsgesellschaft.com/dis/dis.git dis
  14. mDIS sourcecode has been checked out: cd /var/www/dis
  15. (optional) Create a new branch to be independent of master version git switch -c mdis-demo23
  16. Inside the new instance, to install the PHP and JavaScript dependencies.
  17. composer install or composer install --prefer-dist
  18. git add .; git commit -m "composer install";
  19. npm install or npm ci (clean install)
  20. npm run build - first run. just to make sure this command works at all. It will take 2 min. The frontend will not be built properly yet, but this is not a problem. Re-run npm run build after the ./yii migrate commands have been run (Step 64 or so).
  21. (optional) git ignore web/workbox*.js
  22. git add .; git commit -m "npm run build, package-lock.json"
  23. (Optional, as Unixuser mdis); npx update-browserslist-db@latest
  24. (optional) Add the line cd /var/www/dis the file ~/.bashrc, perhaps at the end of the file.
  25. Exit the container as mdis, re-enter as root.
  26. As root, customize some dirs, change ownership to www-data, but leave them still writable to Unixuser mdis: cd /var/www/dis/; chown -R www-data:mdis web/assets/* backend/runtime backend/data/; chmod -R 2775 web/assets/* backend/runtime backend/data/;
  27. Customize the user permissions and group permissions of several key mDIS directories:
  28. Exit the container as root, re-enter as mdis, cd /var/www/dis; chmod -R 2775 backend/dis_templates/ backend/forms backend/models/ src/forms; chown -R www-data:mdis backend/dis_templates/ backend/forms backend/models/ src/forms;
  29. (If this does not work, do it again as root).
    1. git add . ; git commit -m "change file/dir permissions in backend/ and src/ dirs"
  31. Put these directories under version control. Specifically, remove them from the default preexisting .gitignore file. Some important directories (backend/models/, backend/forms/, src/forms/, backend/dis_templates/models, backend/dis_templates/forms) are explicitly not versioned by default, but they should be. Especially when you start adapting the instance: git add .; git commit -m "re-added models/, forms/, /src/forms/, dis_templates/models, dis_templates/forms to version control". So uncomment by removing the # from the respective lines in the .gitignore file. Add web/workbox-* to the .gitignore file.
  32. git add .; git commit --no-verify -m "changed some file modes, permissions and ownerships"
  33. Exit the container as mdis, re-enter as root
  34. (Optional) Upgrade npm to the latest version: npm install -g npm
  35. Create the mysql databases for the new instance: cd /app/mdis-XXX
  36. A LOT OF DETAILS and INSTRUCTIONS ARE MISSING HERE. TODO: add them.
  37. cp ../mdis-SOME-TEMPLATE/deploy-mdis-SOME_INSTANCE.sh .
  38. Search and replace all occurrences of SOME_INSTANCE with the new instance name, e.g. demo23, in the docker-compose.XXX.yml file.
  39. enter new instance container as root, . ./deploy-mdis-demo23.sh
  40. mv ../mdis-demo23-generate-db.sql .
  41. exit the container as root, enter-db
  42. run the sql script from a shell inside the db container. This creates 2 databases and about 5 distinct database users whith random passwords. The passwords were printed to the .sql file. Copy them to secure location. The passwords are used by the mDIS framework to access the database. mysql -u root -p < mdis-demo23-generate-db.sql or use copy-and-paste.
  43. exit the mdis-db container as root,
  44. (optional, on your workstation) check that new datases of the new instance exist with adminer: https://data.icdp-online.org/adminer. They do not containe any tables.
  45. Re-enter mdis-demo23 as root
  46. check file globalname.conf in /etc/apache2/conf-available. Remove duplicate stanzas . Only the last stanza should remain. restart apache. The container will kick you out when apache ends..
  47. Re-enter the container as mdis, cd /var/www/dis.
  48. Check that environment variables are set properly. env | grep MYSQL. This is a quick check that your instance will connect to the correct database: mysql -u$MYSQL_USER -p$MYSQL_PASSWORD -D$MYSQL_DATABASE -e "select 'world' as hello;";
  49. (optional) Run ./yii config/list 2>&1 | more. It successfully connects to the database and then fails (because the database is still empty, no tables exist, including the table with config metadata).
  50. Run migrations: ./yii migrate (or ./yii migrate/fresh --interactive=0). This will setup mDIS tables and the database structure. This will create a lot of text output on the console. If it fails, check the stack traces and fix the problems.
  51. Run ./yii config/list. If everything has been installed properly, and the database connection works, the command should return a list of configuration parameters. All values should be empty by default. If connecting doesn't work, you will get a stack trace. Check the stack trace and fix the problem.
  52. Set at least instance name: ./yii config/set AppShortName mDIS-demo23
  53. Set more properties as you wish. Some are more complicated to set, e.g. the OAuth2 settings.
  54. Try see the login page. Database tables have been created, but not much data has been loaded. The login page should be available at https://data.icdp-online.org/mdis/demo23
  55. Check directory backend/runtime/. There should be new subdirectories cache,dis_templates,logs, mail, created which might have wrong ownerships and permissions. Fix them. Ownership should be www-data:mdis for dirs and files, permissions: 2775 for directories, 775 for files. for d in cache dis_templates logs mail; do chown -R www-data:mdis $d; chmod -R 2775 $d; find $d -type f | xargs -i chmod -R 775 "{}"; done,
  56. Same for /var/www/dis/web/assets:
  57. Run find /var/www/dis/web/assets -type d | xargs -i chmod -R 2775 "{}"; find /var/www/dis/web/assets -type f | xargs -i chmod -R 775 "{}"
  58. There should be a log file app.log. Check it for errors grep error app.log. Many harmless errors (e.g. wrong/incomplete/mistyped ./yii config commands) are expected.
  59. (optional) Check backend/config/web.php file and if necessary adapt the Yii framework (e.g. change its debugging settings, e-mail: change 'class' => 'yii\symfonymailer\Mailer' to class' => 'yii\swiftmailer\Mailer') and the mDIS instance to the new instance.
  60. Run npm run build again. This builds a legacy bundle and a module bundle. Among those is the ArchiveFile.vue file which implements mDIS file upload.
  61. Login with https://data.icdp-online.org/mdis/demo23. Login should be possible with anonymous/anonymous.
  62. Look at the dashboard. Changing the design ("edit mode") is not possible. File-upload is not possible.
  63. (optional) remove the anonymous user from the database, if it is not needed. ./yii user/delete anonymous
  64. (Bugfix) Change the role "administrator" to "administrator" in the backend/commands/SeedController.php file. The role "administrator" has been renamed in the list of predefined roles, but the PHP code expects it to exist. perl -pi -E "s/'_administrator_'/'administrator'/" backend/commands/SeedController.php
  65. Login to Adminer SQL Query Tool to check if the database is populated with tables. https://data.icdp-online.org/adminer. Login with the credentials printed to the mdis-demo23-generate-db.sql file. Check the database mdis_demo23. There should be a lot of tables. Check the table user. There should be a user administrator and an encrypted password. Check the table auth_role. There should be a role with name administrator.
  66. (optional) reverse-engineer the database with an appropriate tool. E.g. with DBeaver or MySQL Workbench. This is useful to understand the database structure (and to issue queries, create new tables etc). Generate an ER diagram and do some layouting and tweaking. Save the ER diagram as an SVG or PNG image and link to it, e.g. from the mDIS instance's home page.
  67. (optional) Login as root. Enable the server-info and server-status Apache handlers. They will display a lot of useful information about the server. a2enconf server-info.conf; service apache2 restart. Password protect them. You now have two diagnostics pages for the server. https://data.icdp-online.org/mdis/XXXX/server-info. Also, server-info and server-status are used by other monitoring tools (e.g. Munin, Nagios) for continuous monitoring of the server.
  68. (optional) Run ./yii help seed. Output:
DESCRIPTION

This commands are used to automatically fill database with data.


SUB-COMMANDS

- seed/add-samples-csv-widget
- seed/copy-list-values-to-new-tables
- seed/create-wr-splits
- seed/example-dump                    Create default tables and records (i.e. "ProjectExpedition")
- seed/form-permissions                Updates the access rights based on the existing forms.
- seed/list-values                     Load mDIS list_values table with more or less useful data
- seed/post-pass                       try to put the password for the "mdis" mysql user on the post-widget
- seed/random-storage-locations
- seed/section-work                    imports some real-world core-section data  into a temp database tables
- seed/templates-files
- seed/templates-tables
- seed/used-value-lists                Import all values list from CSV files in folder 'backend/command/value-lists'
                                       that are used in forms.
- seed/users                           Seed users accounts into database.
- seed/value-lists                     Show and import value lists from CSV files
- seed/widgets                         Load default mDIS widgets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
  1. Run yii seed/XYZ subcommands according to your preferences. Some of these commands might require you to twek some things in the source code. Check directory backend/commands/value-lists/ for example.
  2. Login into the mDIS Frontend, Enter some info to the mDIS forms.