mDIS For System Administrators (SAs)

Note

Many tasks and responsibilites overlap with the mDIS developer role. A clear separation is not always possible.

Related Pages

The list of "Related Rages" is quite long this time 😄. See bottom of page.

This page describes tasks that are usually performed only once, during setup and configuration of mDIS.

mDIS Client-Server Architecture

Most likely, an end user encounters mDIS by opening a web page. See the top of the left side of the following diagram, showing the mDIS implementation technologies and the mDIS client-server architecture:

mDIS from a technical perspective: implementation technologies used.

Web browsers' "various technologies" mentioned above include Local Storage, for instance.

Not mentioned above, but still very important, is the build-system, which is used to compile the mDIS Frontend and to test the Backend.

Generally, software developers and system administrators prefer even more detailed views, which include the mDIS build process, or build system:

mDIS High Level Architecture

Build System and related components

The following figure shows the mDIS Fronted, the mDIS Backend and related tools,such as a low-level database query tool:

The general architecture is shown in another alternative view here:

mDIS implementation, high-level components.

This high-level system design is also known as a "LAMP web service stack" (opens new window).

The most important way to access mDIS is via the webbrowser. However other access mechanisms exist.

The database can be accessed via an SQL API which is conveniently accessible via the adminer.php web page. Command line access via SSH is also possible for system administration purposes.

mDIS Build System

The following figure shows a simplified view of the mDIS build system:

The mDIS Build System, Frontend and Backend

Yii migrations and other CLI commands with the yii tool are also used during the installation and configuration process. (Not shown in the figure.). yii is a lightweight Symfony 5 Console (opens new window) app, and follows conventions used in the wider PHP community.

Frontend

The mDIS frontend, that is what the user sees in the Web-Browser on the client-side, is based on Vue.js (opens new window). On the server-side, mDIS is a PHP7-based Application backed by the Yii2 (opens new window) Framework, the Apache Webserver and a mySQL Database.

Backend

You may decide to use different a Webserver vendor, and a different Database backend. Webserver choices are various (Nginx, Microsoft IIS), and a matter of personal preference. For databases, see what Yii2 (opens new window) supports. You may create an SQLite, MySQL, PostgreSQL, MSSQL or Oracle database, and possibly others via Yii extensions (Informix, IBM DB2, Firebird, MariaDB...).

We develop and test on mySQL 5. 7 (or higher) and Apache 2. 4 (or higher).

For VueJS development, NodeJS is required. Installation of the Vue dependencies will put the Webpack devServer on your development environment.

Webserver configuration

Apache 2.4

In production mode, when running a single mDIS instance on a server, there are no particular nonstandard Apache Modules necessary , except for enabling mod_php (actually, mod_php7 or mod_php8 of course).

If you want to run mDIS via https, you must get an HTTPS certificate, and install that in your webserver. We assume that you know how to adapt some config files yourself. This is not explained here. See extra document and the mDIS Installer scripts for a few hints (opens new window). These documents also mentions the PHP extensions required. See also Section 'PHP 7' below.

Apache mod_rewrite code

Yii2 requires very little URL rewriting.

The directives shown below are pretty basic and standard. Put them into web/.htaccess, or into an appropriate Apache configuration file in /etc/apache2, if you are on Linux. (The yii scaffolding process with command composer create-project --prefer-dist yiisoft/yii2-app-basic basic also creates this web/.htaccess file, among many other files, at project initialisation time). Where you put the following directives depends on your personal preference. (Do not put them in both locations, this can be confusing).

# redirect to index.html by default
DirectoryIndex index.html index.php

# if a directory or a file exists, use it directly
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# otherwise forward it to index.php
RewriteRule . index.php
AddDefaultCharset UTF-8

With URL-rewriting set correctly, you'll be able to run PHP scripts from the web/ directory. You can put your own static HTML pages into web/.

In development mode, a bit more configuration needs to be done, especially if mDIS runs inside Docker containers. This is subject to the personal tastes and preferences of the developers. This is not further described here.

If you run the Adminer database management tool on the same server as mDIS, you should to create an Alias in one of the config files in /etc/apache, or call adminer.php directly.

Installing mDIS side-by-side on a single host

If you want to install mDIS alongside each other on a single host, or with a particular URL-path (such as /mdis/mysite) you can use the Apache Webserver. You must install and enable the Apache Modules "mod_proxy" and "mod_proxy_http" (often thise modules are installed and enabled together though: mod_proxy, mod_proxy_ajp, mod_proxy_balancer, mod_proxy_connect, mod_proxy_html).

If you want to install mDIS such that, for instance...

• the requested URL (handled by the reverse proxy) is "http://dis-app.some-host/mdis/deadsea/"
• and the URL called by the reverse proxy is "http://dis-app.some-host:8111/" (different port, this time two directories deep)

... then use this Snippet in your config file for the Apachae webserver. On Linux, this is usually located in a file dis-app-some-host.conf in the directory /etc/apache2/sites-available/:

ProxyRequests Off
ProxyPass /mdis/deadsea http://dis-app.some-host:8111/dis-app/web
<Location /mdis/deadsea/>
  ProxyPassReverse /dis-app/web
  ProxyHTMLEnable On
  RequestHeader unset Accept-Encoding
  ProxyHTMLURLMap http://dis-app.some-host:8111/mdis/deadsea/ /
  ProxyHTMLURLMap /dis-app/web /
</Location>

TBC TBC TBC

PHP 7

For a complete PHP 7 configuration of a working mDIS installation, see 2019 phpinfo() output. (Click that link twice!)

From this document you should infer which PHP extensions to install, etc.

Page Not Found error - (404 Error)

If a 404 error (Page not found) occurs after clicking on the previous link, simply reload the page. After a second try, this should work and display the familiar phpinfo() output that all PHP developers know and love. The output is an older static snapshot from a development server (not from the data.icdp... server).

Yii UrlManager

For Routing inside the Yii framework, take a look at the urlManager section in backend/config/web.php which defines the main URL Paths that mDIS uses:

'urlManager' => [
            'enablePrettyUrl' => true,
            'enableStrictParsing' => true,
            'showScriptName' => false,
            'rules' => [
                '/' => 'site/index',
                '/test' => 'site/test',
                '/site/error' => 'site/error',
                '<module:(rbac)>/<controller>/<action>' => 'rbac/<controller>/<action>', // role management
                '<module:(user)>/<controller>/<action>' => 'user/<controller>/<action>', // user management
                '<controller:(report)>/<reportName:[\w-]+>' => 'report/generate',        // reports
                '<controller:(files)>/<id:\d+>' => 'files/view',                         // file upload
                '<controller:(files)>/original/<id:\d+>' => 'files/view-original',       // file assignment
                '<controller:(files)>/<action>' => 'files/<action>',
                // csv file importer
                '<controller:(terminal)>/<action>' => 'terminal/<action>',
            ],
        ],

This gives you an overview about the main routes that mDIS defines in the Web layer.

Also look into files backend/modules/api/Module.php and backend/modules/cg/Module.php which contain many important URLManager rules for mapping REST api calls to controller method calls.

Write Permissions

You must give the unixuser owning the webserver unix-process write-permissions to certain directories: (backend/dis_templates/forms/, backend/dis_templates/models/, backend/forms/, backend/models/, src/forms/). See also "for developers" documentation.

Installation with Virtual Box

See page Installation with Virtual Box.

A Vagrant-based installer is also provided (opens new window) as a public repository. Vagrant (opens new window) is a command line utility for managing the lifecycle of virtual machines. e.g. for scripting the setup of VirtualBox VMs. A _Vagrantfile_ is a declarative installation script.

See mDIS-installer README page (opens new window) for details.

Native Installation on Linux, or with Docker

See page Native Installation on Linux.

A Docker configuration will be documented in a later release.

(Docker is a relatively new virtualization software that is free to use but can be complex to configure. It is more suited for software developer workstations.)

Template Manager

The mDIS template manager can be used to configure data entry forms and database tables.

See Extra Pages: Intro, more.

Console Commands

This section will describe some special-purpose commands that are necessary for maintenance of an existing, working mDIS installation.

Examples for these commands are npm run ... or yii migrate/fresh and related.

npm run build

Command npm run build creates a production build of the mDIS frontend, which consists of several *.js- and *.css files, and images. Ultimately npm run build copies these files to a special output directory, web/. Before doing so it deletes older versions of these files from web/. See Developer page.

Directory `web`

Do not delete the contents of the web/ directory manually. Some files put in there might be there for a variety of reasons. E.g. they are static files not served from mDIS, but still important.

Internally, the node package manager npm relies on the bundler webpack (opens new window) to bundle code, transform code, and incorporate assets. Webpack handles approximately the following subtasks:

• Minification, concatenation, code optimizations
• Reasonable node shims to reduce package size
• Chunk hashing for cache busting
• vue-loader for compiling single-file components
• css-loader
• CSS Modules support
• extract-text plugin
• post-css plugin
• sass-loader, stylus-loader and less-loader support
• babel-loader for modern JavaScript support
• eslint-loader
• UglifyJS
• optimize-css-assets
• pre-load plugin
• HtmlWebpackPlugin
• CommonsChunkPlugin
• InlineSourcePlugin

All of this happens automatically, after configuring these build-system components once. Then it requires no input from the mDIS Admin.

Node Module @vue/cli (opens new window) has Webpack built-in as a dev-dependency and calls Webpack. Webpack gets installed when mDIS and Vue are installed. However, the webpack-cli module is not installed by default.

For developers

Custom tasks are not included in the list above. Our custom tasks are:

• injecting the git-tag and the build number into the mDIS dashboard for display.
• creating sourcemaps during a dev build.
• proxy configs for the Webpack dev server (for accessing dockerized backend services)

The relevant config files are vue.config.js and babel.config.js. Presumably Webpack also has more (internal) config files deep inside the node_modules/ directory.

See also directories node_modules/webpack* and node_modules/@vue, node_modules/vue*, and (less so) node_modules/@vue/cli-service/webpack.config.js. The Babel preset is @vue/app (opens new window) which is a specialization of @babel/preset-env (opens new window). Babel (opens new window) is also in directory node_modules.

Run vue ui dashboard from command line with a command similar to this one:

vue ui -H localhost -p 8003 --quiet --headless

Older dev-docs in IG Bremen wiki (opens new window)

TBC

Yii Migrations

About Migrations

Read this section about command line tools first

The console runners section and the migrations section on the developer page.

yii migrate scripts perform initialization tasks and are often needed during mDIS development, especially when you want to setup, reconfigure, or rebuild an mDIS from scratch.

Rebuilding mDIS might be necessary during a running project, e.g. after substantial database schema updates and redesigns; during a new setup of an mDIS; or after all data have been purged, and the system must be rebuilt, i.e. prepopulated with new tables, users, forms, etc. The migration scripts are stored in /backend/migrations.

Forgetting to apply migration tasks can lead to errors. For example, if there were any new columns in any important database tables, and the PHP controller classes can't find these new columns, because the Yii initialization scripts (migrations) were not applied, mDIS will refuse to work properly.

Rebuilding the mDIS database

This is an example of a shell script applying Yii migration scripts, and 4 of our custom seed scripts (custom tasks, related to migrations, another topic in itself).

The following code empties all tables, and deletes all tables and users. Then it rebuilds everything in the correct order, and loads a database-dump with example data, and sets some default permissions.

# Optional: Enter the Docker container, if your mdis runs on with Docker.
# docker exec -it disapp_php_1 bash            # enter container

# this command keeps all data but, but might throw some errors
# only new migrations are applied
# ./yii migrate --interactive=0
#
# this variant of the command drops all existing tables.
# (run this if tables have been redesigned substantially):
./yii migrate/fresh --interactive=0

# additional tasks
./yii seed/users
./yii seed/example-dump
./yii seed/list-values
./yii seed/widgets

./yii seed/form-permissions # probably obsolete, unnecessary

# optional: promote preferred users/logins to _administrator_-role.
# - Pure SQL: execute this inside Adminer
#     update auth_assignment
#     set item_name = '_administrator_'
#     where user_id = select max(id) from user where username = 'superstar';
# TODO: write a migration script for this

The last 3 seed/ tasks are defined in backend/commands/, e.g. in file SeedController.php and in file backend/modules/api/common/controllers/FormController.php, method updateAccessRights().

The ./yii seed/form-permissions task tries to analyze existing PHP Model files and Forms found in the backend directory, created during previous specializations, and tries to assign appropriate edit permissions to the new users just created with seed/users.

The yii script runner is stored at the top level of the mDIS app. Change into this directory and call it as ./yii .... On Windows, use ./yii.bat ....

TBC

yii fix-data tasks

yii seed tasks

yii migrate/fresh tasks

Updating third-party code

Why updating third-party code?

WARNING

Both client-side parts and the server-side components of mDIS make heavy use of open-source third party code. From time to time these code dependencies must be updated. This needs to be done by an administrator.

The upgrade-commands itself are simple, but verifying that upgrades do not break anything can be quite time consuming.

Therefore, major updates should be done cautiously: mDIS admins should try stuff in a development environment first, and then on a staging server (configured similar to a production server, but for internal use only), and then deploy on the production server.

In practice

• The update process requires mDIS having access to the internet. Upgrades without internet access can be done but are cumbersome.
• About ~70% of the mDIS codebase is open-source third-party code. Updates for these libraries are very common, both for Javascipt/Node Modules and the PHP modules that mDIS uses internally. It is the responsibility of the project managers and software developers to come up with a concrete plan and a set of best practices for upgrades. It will vary from project to project.

What needs to be updated can be checked with:

• on the client side: npm outdated
• on the server side: composer outdated
• Linux guest operating system: apt list --upgradable
• Virtual Box GUI: Help menu / Check For Updates
• git mDIS repository: git fetch --dry-run

Updates can be actually performed with:

• on the client side: npm update
• on the server side: composer update
• Linux guest operating system: sudo apt-get upgrade
• Virtual Box GUI: Help menu / Check For Updates, then follow the prompts to install any available updates
• git mDIS repository: git pull

This is explained in greater detail below.

Upgrading Client Side (production environment)

In a production environment, all javascript dependencies are packaged together (by Webpack, by npm run build, started by the mDIS developer) into a single file, web/assets/js/app.<cachebuster-id>.js. This file might load several other chunk*.js files. Together, they occupy very little space, about 2 MB in the web/assets/js directory of the mDIS source-code tree.

For updating or restoring the front end, i.e. the code that runs inside the browser of the mDIS user, it is therefore necessary to copy the contents of the web/ folder from a development machine (or from any storage location) into the web/ folder of the production machine.

Upgrading Client Side (development environment)

The update process of a development instance of the mDIS is much more complicated, because the code-bundling system itself also gets updated. This is described in "For developers".

See also

Developer Documentation: "Advanced System setup and configuration" - Updating Third-party JS Code

Upgrading Server Side (composer)

For the server side, 3rd-party PHP code-dependencies are stored in directory backend/vendor. In March 2019 there were 31 top-level subdirectories in backend/vendor, occupying 270 MB of disk space.

• Basically, upgrade server-side PHP code with
  • composer list
  • composer outdated
  • composer upgrade

Command composer list lists composer commands.

Command composer outdated shows a list of installed packages that have updates available, including their latest version.

Command composer upgrade upgrades the dependencies to the latest version according to composer.json, and updates the composer.lock file (which contains detailed info about the packages actually installed, their repo locations, creation dates, checksums, ...)

Run command composer help for details.

• Verification:
  • see below (Further System Administration)

TODO: describe which versions of 3rd party code are easy to upgrade and which aren't. Describe why some composer packages have their version number fixed in file "composer.json" (and in package.json)

Mysql Administration

The mysql console client

An mDIS admin might want to interact with the mysql/mariadb database server directly.

How?

• can be done with commandline tools mysql (for interactive SQL) and mysqldump (for backups)
• can also use web based admin tool Adminer (opens new window), see also below. Alternatively, use Phpmyadmin or MYSQL Workbench which are more powerful than Adminer, but setting them up is out of the scope of this article.
• check mysql installation directory /var/lib/mysql (as root) and /var/log/mysql (for textbased messagelog/errorlog)

Why?

• can perform backups and restore, independently from mDIS webserver layer
• ideal for incremental backups of science data
• can export data in a range of different formats not supported by mDIS-exporter
• mysql direct access maybe needed some day, e.g. for performance tuning or for moving db data to a different partition or network location

Details: See mysql-backup-and restore page.

Credentials: For VirtualBox instances, see a dot-file in the $HOME directory of the mdis or vagrant unixusers. See also a dot-file in /root.

How to install Adminer inside the virtual machine

Adminer (opens new window) is a web based admin tool for Mysql. Adminer consists of a single big PHP file.

You can simply download Adminer-latest.php and can put this in the web/subdirectory:

curl -sSL http://www.adminer.org/latest.php -o web/adminer.php

Alternatively, run this shell script to install Adminer persistently in the /usr directory:

#!/bin/sh
sudo mkdir /usr/share/adminer/
# download 'adminer.php' 1-page standalone php script
sudo wget "http://www.adminer.org/latest.php" -O /usr/share/adminer/latest.php
sudo ln -s /usr/share/adminer/latest.php /usr/share/adminer/adminer.php
# download 'nette' design = 1 css file
sudo wget -nd https://raw.github.com/vrana/adminer/master/designs/nette/adminer.css -O /usr/share/adminer/adminer.css
# create new apache config file
echo "Alias /adminer.php /usr/share/adminer/adminer.php" | sudo tee /etc/apache2/conf-available/adminer.conf
sudo bash -c 'echo "Alias /adminer.css /usr/share/adminer/adminer.css" >> /etc/apache2/conf-available/adminer.conf'
# load config persistently and restart apache
sudo a2enconf adminer.conf
sudo systemctl reload apache2

Occasionally you might need to update the adminer.php file. You can do so by simply replacing the file in the respective directory:

# use appropriate, more recent version numbers
# (these are from 2018/2019)
mv adminer.php adminer.4.6.2.php
curl -LO https://github.com/vrana/adminer/releases/download/v4.7.3/adminer-4.7.3.php
cp adminer-4.7.3.php adminer.php

The Adminer web interface can do most administrative tasks well. However, it cannot rename MySQL foreign keys, for example. For that you would need a different tool, perhaps phpMyAdmin, or HeidiSQL.

Adminer Plugins

Maybe one of the many free Adminer-Plugins (opens new window) can do the missing task.

We recommend to also install the following Adminer plugins:

With Priority 1: floatThead, suggest-tablefields, restore-menu-scroll. Optional: hideableColumns, dump-json, fk-disable, json-column, stickyColumns, AdminerDumpArray, AdminerDumpMarkdownDict.

restore-menu-scroll: This plugin restores the scroll position of the left (sidebar) menu after a page reload or submitting an SQL query. The plugin needs to be patched to work with the latest Adminer version v4.8.1.

Adminer GUI Tips and Tricks

How to make horizontal scrolling of very wide tables easier:
When you press the Alt Key while clicking on the text of a column head, then this column will disappear from the screen.
Often useful to temporarily remove unnecessary columns from an adminer result page.
Unnecessary columns are, for example, always `NULL, or contain very long text.
(This hack requires the "hideableColumns" plugin to be installed and patched).

Troubleshooting

Troubleshooting measures for beginnners and advanced users.

Frontend problems

• Watch the texts in yellow notification tabs and try to understand their meaning. However, often the yellow balloon is to small to display longer error texts, and it disappears after 10 seconds. Therefore:

• Hit the F12 function key on your keyboard (or CTRL-Shift-I on Windows/Linux, or Command+Option+I on a Mac) to open the console of the Browser's developer-tools. Hit the "Console" tab in the developer-tools window. There the same error should be visible, as in the yellow notification box. Also check the "Network" tab to see if any files are missing, after mDIS unsuccessfully tried to load them in the background. Loading in the background is expected behavior. mDIS does this with an internal component called "Axios". This "axios" name might also appear in the error texts. But errors or missing files should not appear.

• If you are an administrator with access to the backend, you might run npm run build. This creates/transpiles a new frontend. Doing so might delete older cruft. This is a last-resort measure.

• The NodeJS Javascript runtime is required to build the mDIS frontend. NodeJS compiles Vue code to compact Javascript code.

Which NodeJS version?

Q: For development purposes it might be needed to install a recent NodeJS (opens new window) version. However NodeJS is undergoing heavy and rapid development by a community of Javascript developers. So which Node version can you actually use?

A: mDIS was developed and extensively used on Node v8.11 and v8.12. However in 2020 we have switched to Node 12 LTS (Long Term Support), and over the years we have worked with all LTS versions v14, v16, v18, v20.

Backend problems

If you have Shell/Console access to the backend, go to directory backend/runtime/log and check files there. Start with file app.log. Look at the last lines of the file, or any other log file in that directory (they might have different names). These lines contain the most recent errors. The log file is very verbose because it is designed for readability. Information in the log file is spread out over many lines. You might need to go back 200 lines; even if the error happened just moments ago.

To shorten the output, try this shell command: grep error app.log. The output of this command is often informative enough.

Related

1. Installation with Virtual Box
2. Native Installation on Linux, or (more common) with Docker
3. See also this Gitlab Repository (opens new window) in particular this detailed description (opens new window)
4. Using the Template Manager
   • See extra Template Manager page
   • See also "For mDIS Developers" page, and references in there
5. OS-Updates, etc
6. Docker, ICDP-internal with links to even more documents
7. Developers page

Top of page