ICDP Internal: Installation of a new mDIS instance using Docker
ICDP Internal: Installation of a new mDIS instance using Docker
Introduction
As of early 2023, this guide describes how to install a new mDIS instance at ICDP, manually. The installation procedure uses the Docker containerization technology. The installation is done on the ICDP server, and the new instance is accessible from the public internet. This is the most controlled but also the most time-consuming way to install mDIS.
Summary
This guide below describes the manual installation procedure. A "prerequisites" workflow is described in the mDIS-repo wiki on gitlab.gfz-potsdam.de. GitLab's built-in Continuous Integration and Continuous Deployment (CI/CD) features enable this semi-automated installation process.
Also see "mdis-blank" Repo, Container Registry.
There are older, previous versions of this Document: installation-docker and sys-admin-docker-icdp-internal.
| ####Host#### | Task Number | Task |
|---|---|---|
| Linux Desktop PC | 1 | (optional) If you prefer to work locally: Mount remote directory containing config files (.env, docker-compose.yml, etc.) as "shared folder": sshfs rz-vm412.gfz.de:/data1/docker/mdis-images /mnt/rz-vm412/mdis-images/.It is also indispensable to ssh into the Docker Host, rz-vm412. |
| Docker Host (Reverse Proxy) | 2 | After ssh'ing in, cd to the mdis-images directory, or open it in a file manager. |
| Docker Host (Reverse Proxy) | 3 | Copy and paste an appropriate (recent?) existing docker-compose-XXX.yml file from the mdis-images directory.cp docker-compose-XXX.yml docker-compose-<your-instance>.yml. |
| Docker Host (Reverse Proxy) | 4 | In file docker-compose-<your-instance>.yml, <your-instance> is the name of the new instance. If your drilling project is named Plutonia, then, e.g., plutonia would be a good value for <your-instance>. Case-sensitive, use lower-case letters only. |
| Docker Host (Reverse Proxy) | 5 | The image: ... instruction in the docker-compose-<your-instance>.yml file could refer to an existing Docker image, e.g.,git.gfz-potsdam.de:5000/mdis-hub/mdis-php-blank:latest. This is a Debian 11 image, based on the official Docker image yiisoftware/yii2-php:8.3-apache. It also contains a recent version of the mDIS codebase. You can also try an mDIS-foundation image. It contains most of the required and optional PHP 8.3 3rd party packages and a selection of Linux packages needed for mDIS v3, but no mDIS 3rd party JavaScript dependencies and no mDIS code. This image is a bit more low-level but it gives you more control about the mDIS version to install from the source code. |
| Docker Host (Reverse Proxy) | 6 | Search and replace all occurrences of XXX with <your-instance> in the file docker-compose-<your-instance>.yml. |
| Docker Host (Reverse Proxy) | 7 | If you are using a reverse proxy: Assign a new port number to the <your-instance> instance in the docker-compose-<your-instance>.yml file. This is the port number that the reverse proxy uses to access the new instance. |
| Desktop PC | 8 | (optional) On your dev machine ssh rz-vm412.gfz.de; cd /etc/apache2/sites-available. Work as root |
| Docker Host (Reverse Proxy) | 9 | If you are using a reverse proxy: Add a new stanza to the Apache config file. The stanza looks like this: html ProxyPass /<your-instance> https://data.icdp-online.org:Portnumber <Directory>....</Directory> (except that the stanza is longer, the port number is newly assigned.). Details are mentioned in Apache-configuration. |
| Docker Host (Reverse Proxy) | 10 | On data.icdp, create an exchange directory for the new instance: mkdir /data1/docker/mdis-images/<your-instance>. |
| Docker Host (Reverse Proxy) | 11 | Copy or create an .env-mysql file: The file belongs in the directory just created. The file name .env-mysql is freely selectable, but you have to add this name in the docker-compose.<your-instance>.yml file in section services/php_<your-instance>:/env_file:. Change the MYSQL_DATABASE=... entry to the database of the new instance. The file contains the MySQL credentials for the new instance. The credentials will be used by the php/apache process inside the Docker container to access the MySQL database. You can take a peek into .env-mysql of an existing instance, and even copy it cp mdis-XXX/.env-mysql mdis-<your-instance>/, but it is essential to change the database name and the MySQL credentials. (A new empty database will be created later in step ~60). |
| Docker Host (Reverse Proxy) | 12 | (optional) You can also add more environment variables here, (or in a different .env file). Create for example GitLab credentials, or set PHP_ENABLE_XDEBUG=0/1 such that they become environment variables inside the Docker container. |
| Docker Host (Reverse Proxy) | 13 | Check syntax of config file withapachectl configtest;``sudo apachectl restart. will restart Apache on the host. |
| Docker Host (Reverse Proxy) | 14 | Run docker-compose -f docker-compose.<your-instance>.yml --project-name mdis up -d --no-recreate. This pulls the docker image, initializes a new "proto-mDIS" instance. Ignore any warnings about "orphan containers." A new instance should be running, but it is not functional yet. (Service Unavailable). Check basic metadata of container, with `docker ps |
| Docker Host (Reverse Proxy) | 15 | (optional) In the .bash_aliases file on the server, add two new aliases to make entering the new instance easier. The aliases should look like this:alias enter-<your-instance>='docker exec -u mdis -it mdis_php_<your-instance>_1 bash'alias enter-<your-instance>r='docker exec -u root -it mdis_php_<your-instance>_1 bash' |
| Docker Host (Reverse Proxy) | 16 | Run source .bash_aliases to make the aliases available in your shell. |
| Docker Host (Reverse Proxy) | 17 | Run enter-<your-instance>r to enter the new instance as root. Run service apache2 status to check if web server is running inside container. |
| PHP Backend | 18 | (optional) As root, run npm install -g diff-so-fancy (ICDP internal). This installs a syntax highlighting tool useful for git diff commands. |
| PHP Backend | 19 | (optional) Update npm: npm install -g npm. This gets rid of nag messages to update npm. |
| PHP Backend | 20 | Inside the new instance, pwd should return /app; run ls /app to check that the exchange directory with the host has been mounted as a Docker volume. Note that many *.yml files and mdis-* subdirectories are in there. |
| PHP Backend | 21 | Inside the new instance, ls /var/www should return mdis html, and mdis should be empty for "foundation" images, and nonempty for "blank" images. |
| PHP Backend | 22 | chown www-data:mdis /var/www/mdis; chmod 2775 /var/www/mdis. |
| PHP Backend | 23 | (optional) Check that there is a command umask 0002 in /etc/bash.bashrc. Add it if it is missing. It's more permissive than the common umask 0022, as it allows group write access by default. New files will have permissions 664 (rw-rw-r--) and new directories will have permissions 775 (rwxrwxr-x). |
| PHP Backend | 24 | exit the container as root. |
| PHP Backend | 25 | Enter as mdis: enter-<your-instance>. |
| PHP Backend | 26 | Check that MYSQL_* environment variables have been set by Docker, from the .env-mysql file: `env |
| PHP Backend | 27 | (optional) Customize the bash prompt: vi ~/.bash_it/themes/bobby/bobby.theme.bash; change \h to something of your liking, e.g., <your-instance>. Exit/log out of the container as mdis. Re-enter as mdis to see the new prompt (and make sure the new prompt is displayed properly). |
| PHP Backend | 28 | (on foundation image):Run cd /var/www;git clone https://$GITLABUSERNAME:$GITLABPASSWORD@gitlab.informationsgesellschaft.com/dis/dis.git mdis. |
| PHP Backend | 29 | mDIS source code has been checked out: cd /var/www/mdis. |
| PHP Backend | 30 | (optional) Create a new branch to become independent of the master branch git switch -c mdis-<your-instance>. (If there is no .git subdir: git init;) |
| PHP Backend | 31 | Inside the new instance, install the PHP and JavaScript dependencies. |
| PHP Backend | 32 | (on foundation image): composer install or composer install --prefer-dist. |
| PHP Backend | 33 | git add .; git commit -m "Ran composer install";. |
| PHP Backend | 34 | npm install or npm ci (clean install). |
| PHP Backend | 35 | npm run build - first run, just to make sure this command works at all. It will take 2 min. The frontend will not be built properly yet, but this is not a problem. Re-run npm run build after the ./yii migrate commands have been run (Step 64 or so). |
| PHP Backend | 36 | (optional) git ignore "web/workbox*.js". |
| PHP Backend | 37 | git add .; git commit -m "npm run build, package-lock.json". |
| PHP Backend | 38 | (optional, as Unix user mdis); npx update-browserslist-db@latest. |
| PHP Backend | 39 | (optional) Add the line cd /var/www/mdis to the file ~/.bashrc, perhaps at the end of the file. |
| PHP Backend | 40 | Exit the container as mdis, re-enter as root. |
| PHP Backend | 41 | As root, customize some dirs, change ownership to www-data, but leave them still writable to Unix user mdis: cd /var/www/mdis/; chown -R www-data:mdis web/assets/* backend/runtime backend/data/; chmod -R 2775 web/assets/* backend/runtime backend/data/;. |
| PHP Backend | 42 | Customize the user permissions and group permissions of several key mDIS directories: Exit the container as root, re-enter as mdis,cd /var/www/mdis; chmod -R 2775 backend/dis_templates/ backend/forms backend/models/ src/forms; chown -R www-data:mdis backend/dis_templates/ backend/forms backend/models/ src/forms;. (If this does not work, try it again as root). |
| PHP Backend | 43 | git add .; git commit -m "Change file/dir permissions in backend/ and src/ dirs" |
| PHP Backend | 44 | Put these directories under version control. Specifically, remove them from the default preexisting .gitignore file. Some important directories: backend/models/, backend/forms/, src/forms/, backend/dis_templates/models, backend/dis_templates/forms are explicitly not versioned by default, but they should be. Removing those respective lines from the .gitignore file. Add web/workbox-* to the .gitignore file. |
| PHP Backend | 45 | Commit the changes: git add .; git commit -m "Re-added models/, forms/, /src/forms/, dis_templates/models, dis_templates/forms to version control". |
| PHP Backend | 46 | (optional, alternatively): git add .; git commit --no-verify -m "Changed some file modes, permissions and ownerships". |
| PHP Backend | 47 | Exit the container as mdis, re-enter as root. |
| PHP Backend | 48 | sudo apt install gawk for the next steps to work. |
| Docker Host (Reverse Proxy) | 49 | Create the MySQL databases for the new instance:cd /app/mdis-<your-instance>. Either do it manually or use an existing shell script: |
| Docker Host (Reverse Proxy) | 50 | A LOT OF DETAILS and INSTRUCTIONS ARE MISSING HERE. TODO: add them. |
| Docker Host (Reverse Proxy) | 51 | cp ../mdis-SOME-TEMPLATE/deploy-mdis-SOME_INSTANCE.sh .. |
| Docker Host (Reverse Proxy) | 52 | Search and replace all occurrences of SOME_INSTANCE with the new instance name, e.g., <your-instance>, in the docker-compose.<your-instance>.yml file. |
| Docker Host (Reverse Proxy) | 53 | Enter the new instance container as root, . ./deploy-mdis-<your-instance>.sh. |
| Docker Host (Reverse Proxy) | 54 | mv ../mdis-<your-instance>-generate-db.sql .. |
| PHP Backend | 55 | Exit the container as root, enter-db (= docker exec -it mdis_db_1 bash - this enters the database container). |
| PHP Backend or MySQL Container | 56 | Run the SQL script from a shell inside the db container. This creates 2 databases and about 5 distinct database users with random passwords. The passwords were printed to the .sql file. Copy them to a secure location. The passwords are used by the mDIS framework to access the database. mysql -u root -p < mdis-<your-instance>-generate-db.sql or use copy-and-paste. |
| MySQL Container | 57 | Exit the mdis-db container as root. |
| Desktop PC | 58 | (optional, on your workstation) Using a browser, check that database mdis-<your-instance> exists and that some database logins and users exist. With Adminer: https://data.icdp-online.org/adminer. The new database does not contain any tables yet. |
| Docker Host (Reverse Proxy) | 59 | Re-enter mdis-<your-instance> as root. |
| PHP Backend | 60 | (optional): Check file globalname.conf in /etc/apache2/conf-available. Using this file is optional. It can contain a ServerName directive (containerized Apache-webservers complain about missing a good servername), and MYSQL_** env vars. Remove duplicate sections and variables. Only one last section should remain. Restart Apache. The container will kick you out when Apache ends. |
| PHP Backend | 61 | Re-enter the container as mdis, cd /var/www/mdis. |
| PHP Backend | 62 | Check that environment variables are set properly. `env |
| PHP Backend | 63 | (optional) Open .env, replace junk MYSQL Entries with valid values from `env |
| PHP Backend | 63 | (optional) Run `./yii config/list 2>&1 |
| PHP Backend | 64 | Run migrations: ./yii migrate (or ./yii migrate/fresh --interactive=0). This will set up mDIS tables and the database structure. This will create a lot of text output on the console. If it fails, check the stack traces and fix the problems. |
| PHP Backend | 65 | Run ./yii config/list. If everything has been installed properly, and the database connection works, the command should return a list of configuration parameters. All values should be empty by default. If connecting doesn't work, you will get a stack trace. Check the stack trace and fix the problem. |
| PHP Backend | 66 | Set at least the instance name:./yii config/set AppShortName mDIS-<your-instance>; ./yii config/set ShowAppShortName mDIS-<your-instance>; ./yii config/set CopyrightText "© ICDP, 2025". |
| PHP Backend | 67 | Set more properties as you wish. Some are more complicated to set, e.g., the OAuth2 settings. |
| PHP Backend | 68 | Try to see the login page. Database tables have been created, but not much data has been loaded. The login page should be available at https://data.icdp-online.org/mdis/<your-instance>. |
| PHP Backend | 69 | Check directory backend/runtime/. There should be new subdirectories cache, dis_templates, logs, mail, created which might have wrong ownerships and permissions. Fix them. Ownership should be www-data:mdis for dirs and files, permissions: 2775 for directories, 775 for files. `for d in cache dis_templates logs mail; do chown -R www-data:mdis $d; chmod -R 2775 $d; find $d -type f |
| PHP Backend | 70 | Same for /var/www/mdis/web/assets: `find /var/www/mdis/web/assets -type d |
| PHP Backend | 71 | There should be a log file app.log. Check it for errors grep error app.log. Many harmless errors (e.g., wrong/incomplete/mistyped ./yii config commands) are expected. |
| PHP Backend | 72 | (optional) Check backend/config/web.php file and if necessary adapt the Yii framework (e.g., change its debugging settings, e-mail: change 'class' => 'yii\swiftmailer\Mailer' to 'class' => 'yii\symfonymailer\Mailer') and the mDIS instance to the new instance. |
| PHP Backend | 73 | Run npm run build again. This builds a legacy bundle and a module bundle. Among those is the ArchiveFile.vue file which implements mDIS file upload. |
| PHP Backend | 74 | Login with https://data.icdp-online.org/mdis/<your-instance>. Login should be possible with anonymous/anonymous. |
| PHP Backend | 75 | Look at the dashboard. Changing the design ("edit mode") is not possible. File upload is not possible. |
| PHP Backend | 76 | (optional) Remove the anonymous user from the database, if it is not needed. ./yii user/delete anonymous. |
| PHP Backend | 77 | (Bugfix) _administrator_ to administrator in the backend/commands/SeedController.php file. The role "administrator" has been renamed in the list of predefined roles, but the PHP code expects it to exist.perl -pi -E "s/'_administrator_'/'administrator'/" backend/commands/SeedController.php. |
| Desktop PC or any host | 78 | Login with SQL Client, e.g, Adminer, to check if the database is populated with tables. https://data.icdp-online.org/adminer. Login with the credentials printed to the mdis-<your-instance>-generate-db.sql file. Check the database mdis_<your-instance>. There should be a lot of newly created tables. Check the table user. There should be a user anonymous and an encrypted password. Check the table auth_role. There should be a role with the name anonymous. |
| Desktop PC | 79 | (optional) Reverse-engineer the database with an appropriate tool and generate an ER diagram. E.g., with DBeaver or MySQL Workbench. This is useful to understand the database structure (and to issue queries, create new tables, etc.). Edit your ER diagram, do some layouting and labeling. Save the ER diagram as an SVG or PNG image and link to it, e.g., from the mDIS instance's home page. mDIS-Blank Diagram |
| Docker Host (Reverse Proxy) | 80 | (optional) Login as root. Enable the server-status Apache handlers. It will display a lot of useful information about current activity on the server. a2enmod status; service apache2 reload. Password protect them because they contain secrets. You now have a diagnostics page for the server. https://data.icdp-online.org/mdis/XXXX/server-status. which is used by external monitoring tools (e.g., Munin, Nagios) for continuous monitoring of the server. |
| PHP Backend | 81 | (optional) Run ./yii help seed. Output:[^seed1] |
| PHP Backend | 82 | Run yii seed/XYZ subcommands according to your preferences. Some of these commands might require you to tweak some things in the source code. Check directory backend/commands/value-lists/ for example. Run `ls -1 backend/commands/value-lists/ |
| PHP Backend | 83 | Login into the mDIS Frontend, enter some info into the mDIS forms. |
| Docker host | 84 | Copy permission-change script from /app/fix* into icdp_work |
| External Server (wb31) | 85 | Add new instance to monitoring jobs, backup jobs,permission-change jobs, etc. |
| Docker host | 86 | Check if directory var/www/dis really is located on a persistent volume. (/data1/.../_data_mdis_<your_instance>_web should be nonempty) |
| Database host | 87 | (optional) in db dba_db, open view users_loggedin_codegen, select all values from column Name_exp_2 and update view definition users_loggedin_mdis (remove the last UNION ALL) |
Footnotes
[^seed1]: Some details on how the "yii seed" command populates the database. Each of these tasks is optional. Therefore they have moved out of the main "database migrations" installation procedure.
Output of yii help seed command:
DESCRIPTION This command is used to automatically fill the database with data.
SUB-COMMANDS
RELEVANT COMMANDS ONLY!
- seed/list-values Load mDIS list_values table with more or less useful data
- seed/post-pass Try to put the password for the "mdis" MySQL user on the post-widget
- seed/random-storage-locations
- seed/section-work Imports some real-world core-section data into temp database tables
- seed/used-value-lists Import all values list from CSV files in folder 'backend/command/value-lists' that are used in forms.
- seed/users Seed user accounts into the database.
- seed/value-lists Show and import value lists from CSV files
Other installation methods
VirtualBox: basics, from online instance, mDIS-installer Repo:.