mDIS User Documentation

Further System Administration

System Administration)

Configuration management

HTTP and HTTPS connectivity

HTTPS enables secure login and secure data transfer.
HTTPS connectivity for mDIS can be achieved with a set of free tools and services, as explained below.
However, at this time, only insecure HTTP is supported out-of-the-box. HTTPS needs some extra work done by the mDIS administrator.

HTTPS with LetsEncrypt.org

When installing and configuring HTTPS support for mDIS with certificates from LetsEncrypt.org, the mDIS instance must be connected to the public internet at least once, to automatically sign the server certificate.
Certificate renewal is necessary every three months.

Why is HTTPS not enabled by default? Because HTTPS support requires a unique server certificate for each mDIS instance.

As a simple workaround, you can create a self-signed server-certificate. However, your browsers will recognize that the sertificate is self-signed, and will complain, because self-signed certificate are not completely trustworthy.

As another workaround, the non-profit organisation Letsencrypt.org (opens new window) issues these server- certificates for free.
However, we cannot package mDIS with such a free LetsEncrypt certificate. These certificates have a default lifetime. They expire after 3 months (unless renewed). Thus, they are not meant to be redistributed freely.

Each newly configured mDIS intance (if run standalone on its own host) needs to get a certificate separately. Hence we cannot simply put an older certificate inside the mDIS installation package.

When mDIS is installed, mDIS needs to be reachable from the public internet at least once, to register the certificate with LetsEncrypt.org. Moreover, the certificate needs to be renewed every 3 months. The renewal process can be automated, though. Then, the LetsEncrypt "Certbot" software automatically renews the certificate, if it is configured to do so. Here is a snippet from the update log:

2020-09-29 09:28:51,291:DEBUG:certbot.renewal:no renewal failures
2020-09-29 20:16:23,840:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2020-10-29 13:42:31 UTC.
2020-09-29 20:16:23,840:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2020-09-29 20:16:25,354:INFO:certbot.main:Renewing an existing certificate
2020-09-29 20:16:34,787:DEBUG:certbot.storage:Writing new config /etc/letsencrypt/renewal/rundis.com.conf.new.
2020-09-29 20:16:34,978:DEBUG:certbot.renewal:no renewal failures
2020-09-30 07:50:18,756:INFO:certbot.renewal:Cert not yet due for renewal
2020-09-30 07:50:18,759:DEBUG:certbot.renewal:no renewal failures
1
2
3
4
5
6
7
8

To see the metadata of the new certificate, refresh your browser, then click on the little lock icon at the top of the page, in the URL address bar.

Domain Names

If mDIS runs standalone on the internet, you need to give the host a humanreadable hostname such as myproject.mdis.org. HTTPS will not work with hostnames that are reachable by IP address such as 127.0.0.1.

Use these free services to register a domain name for mDIS:

freedns (opens new window): Paid & No-Charge Domain Hosting & DNS Hosting Services. (Many others such services exist).

LetsEncrypt (opens new window): A nonprofit Certificate Authority providing TLS certificates to more than 200 million websites.
CertBot (opens new window): Certbot is a free, open source software tool for automatically using Let’s Encrypt certificates on manually-administrated websites to enable HTTPS. Install and run this software on your mDIS.

  • if you need encryption,
  • if you want to run mDIS standalone on the public internet, and
  • if you do not have alternative in-house operating-procedures to sign HTTPS-Server certificates. (Many institutes and organisations have IT staff responsible for doing this and do not use LetsEncrypt.)

SSL Server Test (opens new window): An optional service to list all properties of LetsEncrypt certificates, to perform a quality check if your webserver is configured to serve HTTPS correctly.

(TBC)

PWAs -Progressive Web App

HTTPS is also needed for Service Workers (opens new window), a key component of PWAs, Progressive Web Apps. PWAs deliver a nice user experience on smartphones. Service workers make mDIS run better offline, by caching the content of mDIS pages, and by changing the look of the webbrowser in a specific way. The mDIS already has PWA features built-in, but they are disabled when HTTPS is not available.

There has not been much demand for the PWA/Service worker functionality, so it is a bit under-developed.

TBC

Development and staging

A mDIS admin should maintain a development server where to verify the upcoming software updates, and to try out new things.

This could be an own development machine, but ideally the mDIS SA also maintains a staging virtual machine. That is a setup that is more or less identical to the Virtual machine that is used in production, i.e. during an ongoing drilling project.

OS Updates

Updates of the guest OS inside the virtual machine

  • Most Linux distributions provide security updates and install these automatically. For the LTS (long time support) version of Ubuntu (like 18.04 LTS), security updates are provided for 5 years. After that you have to upgrade to a newer Ubuntu version. The Ubuntu developers provide a migration tool to do this.

  • On Ubuntu automatic security updates are configured by default (on server and desktop version). You just have to apply them:

  • If a Linux is used as a guest OS, use the self-upgrading software used by the operating system. You need to be Unixuser root or belong to the "sudo" group to do this (find out with shell-command sudo -l -U YOURUSERNAME).

    • For Ubuntu Linux, run apt list --upgradable to see packages that can be updated. Run apt update && apt upgrade, or use the GUI tool "Software Updater" for managing updates.
    • If you want to prevent updates for specific packages only, mark them as untouchable; in Ubuntu/Debian use "apt-mark hold <package>" or similar (opens new window)
    • Make sure that update-grub script can do its work properly: There must be one partition marked as bootable. [TBC], otherwise -sooner or later- the apt upgrade command will create a weird confirmation dialog asking "Into which partition should update-grub write the boot record?"

Updates of the Host operating system

For a permanent installation, the server administrator has to decide, if or when to update software versions. You should apply the security updates for the host operating system.

  • Windows VM-Hosts. Updates for the host operating system have to be applied for* security reasons, especially if it is Microsoft Windows.

  • Linux VM-Hosts. Minor security updates are applied automatically, or you will receive periodic reminders to manually update, if the Linux OS has an internet connection.

Recommendation: No major upgrades of the host OS

For a project installation (on a laptop), we recommend to not manually update or upgrade the system to a new major release, e.g. Windows 8 to Windows 10, or Ubuntu 18.04 to Ubuntu 20.04. If you have to upgrade, make a full backup of every host on your mDIS network (e.g., if you run the mySQL Database on its own host) before such a step. More importantly, export your VM to a fresh .ova file first. At least take a VirtualBox snapshop.

For the guest VM, security updates are optional. Major version updates should be performed on a test installation first, or not at all.

For any upgrade holds: it should not affect the VirtualBox installation. If Windows or VirtualBox does not work anymore after an update, but you still have a recent *.ova file, or the hard disk file ("*.vdi"), you can continue working with the mDIS installation on the same or a different computer without losing any data.

Updates of VirtualBox itself

There is normally no need to upgrade the Virtualbox application. Turn off the update-checks in the VirtualBox GUI, or set notification intervals to the maximum value.

Recommendation: Do not update the VirtualBox Application

For a project installation (on a laptop), we recommend: do not update the VirtualBox application.

Verification of software upgrades

It is important to verify that software upgrades did work properly.

General verification strategy:

  • check output emitted during update process
  • run the client-side test suites (npm run test:unit, npm run test:e2e) TBC
  • run mDIS interactively and check the Browser Devtools Console (CTRL+SHIFT+K)
  • check logfiles during mDIS startup.
    • mDIS logiles are in backend/runtime/logs/.
    • On Linux Guests: Inside Linux guests the kernel startup messages of the last boot can be seen with dmesg -T. Various operating system logs are inside directory /var/log/, but often you need to be root to see them. If the guest OS runs systemd, most recent errors can be seen with journalctl -xe -p err --since today. For a GUI tool, call gnome-logs, installed in dir /snap/gnome-logs.

File-Upload

File-Upload as VirtualBox Shared Folder

Accessing the Upload directories directly

There can be additional ways to access the upload directory /backend/data/upload as a folder of the hard disk on the host: In the VirtualBox appliance containing the Backend, this folder is mapped to a directory on the host computer, so you can easily bulk-copy files into this directory. The mDIS file importer will detect directory changes and try to import files automatically.

Remote upload folders: On permanent server installations there could be a remote folder mapped to the shared folder. For instance, on a Windows host system, a Windows Shared Folder could represent the upload directory. (On Linux, the Shared Folder could be mapped to a Samba share, to an FTP server directory mounted via FUSE filesystem, to an NFS mount, etc. We never tried this.).

Image/Picture conversion

mDIS has a file-uploading feature. The most important use case is uploading pictures of core scans. See Viewer-Operator, section "File Upload".

To use this with all features (e.g. preview-image generation), the free software Imagemagick (opens new window) must be installed on the webserver. The Imagemagick command line tools (convert, identify) must be available to the webserver process such that PHP's exec() function can call these executables. (PHP module imagick does not have to be installed.)

An mDIS operator uses these tools to create screen-sized JPEG Files and JPG thumbnails from oversized, high-resolution corescan pictures.

Importing CSV files

A CSV importer is located in backend/importers/CsvImporter.php. Usage: See Viewer-Operator documentation for the basics, and section File-Importers for technical details.

A related topic is importing table- and tableset definitions, discussed in the next paragraphs.

Transferring items between mDIS installations

Sometimes an mDIS admin will need to transfer some item, such as a preconfigured input form, from mDIS installation A to mDIS installation B. This can be done with the Templates-Manager.

mdis-upload

The sidebar (greyish blue in the figure below) comprises three file-upload textfields (those with a dashed blue line) where the user can drag-and-drop *.json files into. These are now "registered" on the new mDIS (system B), but the changes must still be applied. After import, you still need to create the new data-tables and forms respectively, by clicking the "Save and Generate" Button.

The form can also import .zip File Archives, which have been exported from another mDIS. The zip-Files would contain both backend code (.php, .json files) and frontend code (.vue file).

The whole operation has a lot of issues and shortcomings. It should onbly be done by experienced mDIS administrators, and the table-definitions to be uploaded should be compatible with those already existing in the current mDIS instance.

The backend/config directory

web.php

In the backend/config directory you can find important information about many fine points on how the Yii2 framework was customized for mDIS.

See in particular the config/web.php configuration file.

This file is also critically important to enable mDIS to send emails, e.g. when a new user was registered, or an existing user wishes to change the password.

More about email configuration

The backend/vendor Directory

The design of the server backend, i.e. the Yii2 app, closely follows the "Yii 2 Basic Project Template", with package kartik-v/yii2-mpdf added as an extra dependency.

The Yii2 main framework is installed, see directory /backend/vendor/yiisoft, and the Yii extensions (opens new window) installed separately by Composer are currently (Oct 2019): yii2-authclient, yii2-bootstrap (Twitter Bootstrap 3.4.x), yii2-composer, yii2-debug, yii2-faker, yii2-gii, yii2-httpclient, yii2-swiftmailer.

Better check yourself. Have a look at configfile backend/vendor/yiisoft/extensions.php and check which Yii extensions are actually being loaded.

PHP Dependencies List

See how PHP Dependencies have changed from 2019 to 2022.

HTML 5 APIs

Local Storage

Why is it used in mDIS, what's in there?
Some stored user-data, e.g. mDIS filter Settings, pages marked as Favorites, needs to be saved across browser sessions. This is what localStorage is for. Being built into modern browsers, localStorage is similar to sessionStorage, except that while data stored in localStorage has no expiration time.
To have a look at Local Storage data interactively, open the Browser Development Tools (With F12 key or Shift+CTRL+I).

localstorage-annotated

You have to click Tab "Storage" (1) then expand the "Local Storage" folder in the devtools-sidebar (2), and then you can see the current entries for the URL currently active. The entries have a key-value structure (3).

With a context menu you can alter or delete the items. (Right Mouse Click on a key of interest).

TBC: How to change it programmatically (= more selectively)?

more HTML5 APIs

TBC - Service Worker (see "PWA", above).

developer page

More... TBC

Still To Be Documented:

  • Effect of Software Firewalls, Antivirus-Software on OS - ??? (TBC) (likely to be important for troubleshooting)

  • Effect of Adblockers, Do-Not-Track settings, JS permissions, JS same-host-origin-policy of Browsers (all not very important for mDIS)