Further System Administration

Related Pages

System Administration

Configuration management

HTTP and HTTPS connectivity

HTTPS enables secure login and secure data transfer.
HTTPS connectivity for mDIS can be achieved with a set of free tools and services, as explained below.
However, at this time, only insecure HTTP is supported out of the box. HTTPS needs some extra work done by the mDIS administrator.

Why is HTTPS not enabled by default? Because HTTPS support requires a unique server certificate for each mDIS instance.

As a simple workaround, you can create a self-signed server certificate. However, your browsers will recognize that the certificate is self-signed and will complain because self-signed certificates are not completely trustworthy.

As another workaround, the non-profit organization Letsencrypt.org issues these server certificates for free.
However, we cannot package mDIS with such a free LetsEncrypt certificate. These certificates have a default lifetime. They expire after 3 months (unless renewed). Thus, they are not meant to be redistributed freely.

Each newly configured mDIS instance (if run standalone on its own host) needs to get a certificate separately. Hence we cannot simply put an older certificate inside the mDIS installation package.

When mDIS is installed, mDIS needs to be reachable from the public internet at least once to register the certificate with LetsEncrypt.org. Moreover, the certificate needs to be renewed every 3 months. The renewal process can be automated, though. Then, the LetsEncrypt "Certbot" software automatically renews the certificate if it is configured to do so. Here is a snippet from the update log:

2020-09-29 09:28:51,291:DEBUG:certbot.renewal:no renewal failures
2020-09-29 20:16:23,840:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2020-10-29 13:42:31 UTC.
2020-09-29 20:16:23,840:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2020-09-29 20:16:25,354:INFO:certbot.main:Renewing an existing certificate
2020-09-29 20:16:34,787:DEBUG:certbot.storage:Writing new config /etc/letsencrypt/renewal/rundis.com.conf.new.
2020-09-29 20:16:34,978:DEBUG:certbot.renewal:no renewal failures
2020-09-30 07:50:18,756:INFO:certbot.renewal:Cert not yet due for renewal
2020-09-30 07:50:18,759:DEBUG:certbot.renewal:no renewal failures

To see the metadata of the new certificate, refresh your browser, then click on the little lock icon at the top of the page, in the URL address bar.

Domain Names

If mDIS runs standalone on the internet, you need to give the host a human-readable hostname such as myproject.mdis.org. HTTPS will not work with hostnames that are reachable by IP address such as 127.0.0.1.

Use these free services to register a domain name for mDIS:

freedns: Paid & No-Charge Domain Hosting & DNS Hosting Services. (Many other such services exist).

LetsEncrypt: A nonprofit Certificate Authority providing TLS certificates to more than 200 million websites.
CertBot: Certbot is a free, open-source software tool for automatically using Let’s Encrypt certificates on manually administrated websites to enable HTTPS. Install and run this software on your mDIS.

• if you need encryption,
• if you want to run mDIS standalone on the public internet, and
• if you do not have alternative in-house operating procedures to sign HTTPS server certificates. (Many institutes and organizations have IT staff responsible for doing this and do not use LetsEncrypt.)

SSL Server Test: An optional service to list all properties of LetsEncrypt certificates, to perform a quality check if your web server is configured to serve HTTPS correctly.

(TBC)

PWAs - Progressive Web App

HTTPS is also needed for Service Workers, a key component of PWAs, Progressive Web Apps. PWAs deliver a nice user experience on smartphones. Service workers make mDIS run better offline, by caching the content of mDIS pages, and by changing the look of the web browser in a specific way. The mDIS already has PWA features built-in, but they are disabled when HTTPS is not available.

There has not been much demand for the PWA/Service worker functionality, so it is a bit underdeveloped.

TBC

Development and staging

A mDIS admin should maintain a development server where to verify the upcoming software updates and to try out new things.

This could be an own development machine, but ideally, the mDIS SA also maintains a staging virtual machine. That is a setup that is more or less identical to the virtual machine that is used in production, i.e., during an ongoing drilling project.

OS Updates

Updates of the guest OS inside the virtual machine

• Most Linux distributions provide security updates and install these automatically. For the LTS (long-term support) version of Ubuntu (like 18.04 LTS), security updates are provided for 5 years. After that, you have to upgrade to a newer Ubuntu version. The Ubuntu developers provide a migration tool to do this.

• On Ubuntu, automatic security updates are configured by default (on server and desktop versions). You just have to apply them:

• If a Linux is used as a guest OS, use the self-upgrading software used by the operating system. You need to be Unix user root or belong to the "sudo" group to do this (find out with shell command sudo -l -U YOURUSERNAME).

  • For Ubuntu Linux, run apt list --upgradable to see packages that can be updated. Run apt update && apt upgrade, or use the GUI tool "Software Updater" for managing updates.
    • If you do not want any updates, open the "Software & Updates" app and disable update checks (See Detailed easy tutorial).
  • If you want to prevent updates for specific packages only, mark them as untouchable; in Ubuntu/Debian use "apt-mark hold <package>" or similar
  • Make sure that the update-grub script can do its work properly: There must be one partition marked as bootable. [TBC], otherwise -sooner or later- the apt upgrade command will create a weird confirmation dialog asking "Into which partition should update-grub write the boot record?"

Updates of the Host operating system

For a permanent installation, the server administrator has to decide if or when to update software versions. You should apply the security updates for the host operating system.

• Windows VM-Hosts. Updates for the host operating system have to be applied for *security reasons, especially if it is Microsoft Windows.

• Linux VM-Hosts. Minor security updates are applied automatically, or you will receive periodic reminders to manually update if the Linux OS has an internet connection.

For the guest VM, security updates are optional. Major version updates should be performed on a test installation first, or not at all.

For any upgrade holds: it should not affect the VirtualBox installation. If Windows or VirtualBox does not work anymore after an update, but you still have a recent *.ova file, or the hard disk file ("*.vdi"), you can continue working with the mDIS installation on the same or a different computer without losing any data.

Updates of VirtualBox itself

There is normally no need to upgrade the VirtualBox application. Turn off the update checks in the VirtualBox GUI, or set notification intervals to the maximum value.

Verification of software upgrades

It is important to verify that software upgrades did work properly.

General verification strategy:

• check output emitted during the update process
• run the client-side test suites (npm run test:unit, npm run test:e2e) TBC
• run mDIS interactively and check the Browser Devtools Console (CTRL+SHIFT+K)
• check log files during mDIS startup.
  • mDIS log files are in backend/runtime/logs/.
  • On Linux Guests: Inside Linux guests, the kernel startup messages of the last boot can be seen with dmesg -T. Various operating system logs are inside the directory /var/log/, but often you need to be root to see them. If the guest OS runs systemd, most recent errors can be seen with journalctl -xe -p err --since today. For a GUI tool, call gnome-logs, installed in dir /snap/gnome-logs.

File-Upload

File-Upload as VirtualBox Shared Folder

Image/Picture conversion

mDIS has a file-uploading feature. The most important use case is uploading pictures of core scans. See Viewer-Operator, section "File Upload".

To use this with all features (e.g., preview-image generation), the free software Imagemagick must be installed on the web server. The Imagemagick command line tools (convert, identify) must be available to the web server process such that PHP's exec() function can call these executables. (PHP module imagick does not have to be installed.)

An mDIS operator uses these tools to create screen-sized JPEG files and JPG thumbnails from oversized, high-resolution core scan pictures.

Importing CSV files

A CSV importer is located in backend/importers/CsvImporter.php. Usage: See Viewer-Operator documentation for the basics, and section File-Importers for technical details.

A related topic is importing table and tableset definitions, discussed in the next paragraphs.

Transferring items between mDIS installations

Sometimes an mDIS admin will need to transfer some item, such as a preconfigured input form, from mDIS installation A to mDIS installation B. This can be done with the Templates Manager.

The sidebar (greyish blue in the figure below) comprises three file-upload text fields (those with a dashed blue line) where the user can drag-and-drop *.json files into. These are now "registered" on the new mDIS (system B), but the changes must still be applied. After import, you still need to create the new data tables and forms respectively, by clicking the "Save and Generate" button.

The form can also import .zip file archives, which have been exported from another mDIS. The zip files would contain both backend code (.php, .json files) and frontend code (.vue file).

The whole operation has a lot of issues and shortcomings. It should only be done by experienced mDIS administrators, and the table definitions to be uploaded should be compatible with those already existing in the current mDIS instance.

The backend/config directory

web.php

In the backend/config directory, you can find important information about many fine points on how the Yii2 framework was customized for mDIS.

See in particular the config/web.php configuration file.

This file is also critically important to enable mDIS to send emails, e.g., when a new user was registered, or an existing user wishes to change the password.

More about email configuration

The backend/vendor Directory

The design of the server backend, i.e., the Yii2 app, closely follows the "Yii 2 Basic Project Template," with package kartik-v/yii2-mpdf added as an extra dependency.

The Yii2 main framework is installed, see directory /backend/vendor/yiisoft, and the Yii extensions installed separately by Composer are currently (Oct 2019): yii2-authclient, yii2-bootstrap (Twitter Bootstrap 3.4.x), yii2-composer, yii2-debug, yii2-faker, yii2-gii, yii2-httpclient, yii2-swiftmailer.

Better check yourself. Have a look at config file backend/vendor/yiisoft/extensions.php and check which Yii extensions are actually being loaded.

PHP Dependencies List

See how PHP Dependencies have changed from 2019 to 2022.

HTML 5 APIs

Local Storage

Why is it used in mDIS, what's in there?
Some stored user data, e.g., mDIS filter settings, pages marked as favorites, needs to be saved across browser sessions. This is what localStorage is for. Being built into modern browsers, localStorage is similar to sessionStorage, except that while data stored in localStorage has no expiration time.
To have a look at Local Storage data interactively, open the Browser Development Tools (With F12 key or Shift+CTRL+I).

You have to click Tab "Storage" (1) then expand the "Local Storage" folder in the devtools sidebar (2), and then you can see the current entries for the URL currently active. The entries have a key-value structure (3).

With a context menu, you can alter or delete the items. (Right Mouse Click on a key of interest).

TBC: How to change it programmatically (= more selectively)?

• See MDN: "Local Storage" inside the Browser. Full example on MDN.

more HTML5 APIs

TBC - Service Worker (see "PWA," above).

developer page

More... TBC