Converting an online mDIS instance to VirtualBox instance

WORK IN PROGRESS

In this article, "Online instance" means "mDIS running as a Docker container on a remote server".

Why convert an online instance to a VirtualBox instance?

This might become necessary:

If you want to use the instance somewhere where there is no internet, such as during fieldwork in remote locations.
Another scenario is that you want to make changes to your own local copy of the instance without affecting the online version.
Another scenario is you want to reduce IT security measures for a local copy of the instance, e.g., to allow for more convenience in development and for everyday users.

Note

These instructions are ICDP-OSG-specific. They assume that you have access to the online instance and that you have the necessary permissions to access all parts of its infrastructure (e.g., the network). They also mention some very specific details such as certain hostnames and paths. You will need to adapt these instructions to your own situation.

At a glance

Strategy

Find a recent backup of the online instance (You'll need an SQL-DB backup and a PHP backend backup).
Download the backups to your local machine.
Install VirtualBox and Vagrant on your local machine, if you haven't already. Read this guide
Prepare a Linux Image for the guest VM. Manually convert a certified Cloud Image (e.g., from Ubuntu) into a Vagrant-compatible box, use it locally for mDIS installation. Alternatively Vagrant can download an preconfigured image autonomously from Vagranthub in step 8.
Become a Vagrant expert. Then find the Vagrant provisioning scripts in charge of installation subtasks (file-copying and shell script executions). Look at those here
Create a new Vagrantfile for the VirtualBox instance. Or adapt an existing Vagrantfile via copy-paste and editing.
Adapt further configuration files, e.g., the .env-... files and other ".dotfiles".
Execute the Vagrantfile with vagrant up. This will create the new VirtualBox VM.
Do fine-tuning of Linux guest, the Vagrantfile and the provisioning scripts. Loop between steps 6-9 until both the guest and host run smoothly. Finetune the new guest OS and its graphical desktop (if you installed one).
Do fine-tuning of the mDIS instance itself. Work with mDIS forms as you usually would.
Export the VirtualBox instance to an .ova file if you want to share it with others.

Step-by-step instructions

These instructions show in greater detail how to convert an online mDIS instance to a VirtualBox instance.

Steps 1-10 are preparatory steps.

Find a recent backup of the online instance.
- Alternatively, create a new backup of the online instance.
- Download the backup to your local machine.
  - The backup is a .zip file for the SQL database dump.
  - For the PHP-based backend code, you can copy the entire Docker volume into a zipfile on your local machine. This creates a most recent copy of the entire PHP backend code.
Download the backups to your local machine.
- Command: scp -r user@remote:/path/to/backup /path/to/local/backup
- For wb33 PC:
  - Fetch mDIS Database dump:

#!/usr/bin/env bash
# This script fetches the most recent backup of the mDIS database from a backup_server
# call as: download-recent-mdis-backup.sh mdis_goedeep
current_date=$(date +%Y-%m-%d -d "yesterday")
instance_name=$1 #"your_instance_name"
backup_server=somehost.gfz.de
filename_prefix=mysqldb_backup_data.icdp-online.org
if [ -z "$instance_name" ]; then
    echo "Please provide the instance name  the first argument, "
    echo "(as it appears in backup files, e.g. mdis_demo25)"
    exit 1
fi
# Copy from server which stores DB backups to local machine
dump_zip_dir=/data2/backup/rz-vm412/mdis-sqldumps
dump_zip_file=${filename_prefix}--${current_date}--${instance_name}.sql.zip
local_dump_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/db-dumps
scp -r $backup_server:$dump_zip_dir/$dump_zip_file $local_dump_dir
echo "now continue here:"
echo "cd $local_dump_dir"

#local_dump_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/db-dumps
cd $local_dump_dir;  
ZIPFILE=${filename_prefix}--${current_date}--${instance_name}.sql.zip
SQLFILE=${filename_prefix}--${current_date}--${instance_name}.sql
unzip -l $ZIPFILE        # have a look at the contents of the zip file
unzip $ZIPFILE $SQLFILE  # extract the sql file

ls -lrt  $local_dump_dir | tail -n 4

fetch mDIS Docker Volume, which contains the entire /var/www/dis directory.
(expect to download a few 100 MB for new/upcoming projects, but expect many GBs for long-running projects). Even larger in size when downloading without zip compression, uncompressed. Check your disk space before downloading and decompressing.

#!/usr/bin/env bash
# This script fetches the most current state of an mDIS docker volume from a production server
instance_name=$1 #"your_instance_name"
if [ -z "$instance_name" ]; then
    echo "Please provide the instance name  the first argument, "
    echo "(as it appears in backup files, e.g. mdis_demo25)"
    exit 1
fi
# Copy from server which stores DB backups to local machine
current_date=$(date +%Y%m%d -d "yesterday")
mdis_host=somehost.gfz.de
volume_dir=/data1/docker/docker-data/volumes/${instance_name}_web/_data  

local_zip_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/${instance_name}-${mdis_host}
mkdir -p "$local_zip_dir"
cd "$local_zip_dir"
ssh $mdis_host "cd $volume_dir && zip -r - ." > "${instance_name}"-fe-be-"${current_date}"-"${mdis_host}".zip

In the above ssh command, the content of the prospective zip file is not saved to a temporary zip file on the remote machine. Instead, it is created and streamed directly over the network using an in-memory buffer. The zip command outputs the zipped content to stdout, which is then piped through the SSH connection to the local machine. This approach avoids the need for temporary storage (of a potentially very large zip file) on the remote machine.
Summary so far: This created a zipped local copy of the current state of the online instance. 2 Files were downloaded to the local machine:
- mysqldb_backup_data.icdp-online.org--2024-05-02--mdis_demo25.sql.zip (a few MB in size)
- mdis-demo25-fe-be-20240502-rz-vm412.zip (here ~500 MB compressed data).
- Optional: You can unzip the big Frontend-Backend (fe-be) dump locally after transfer, remove some directories that you do not need, and re-zip it. This will save space on your local machine. If you re-zip, do not forget to zip dotfiles and "dot-directories" as well.
- For example, Files in directories backend/vendor and frontend/node_modules containing 3rd-party dependencies are also included in the backup. You may delete those. Also some of them might need to get recompiled in your new Vagrant box (for example, .c files in the node_modules/ directory might need recompilation). This depends on your VM image and the versions of the dependencies you require. If your machine is similar to the source machine, you might not need to recompile. This is a complex topic not discussed in this guide.

Unzip the database SQL file on your local machine
- Commands:

   # see above,
   # script: download-recent-mdis-backup.sh mdis_goedeep
   current_date=$(date +%Y-%m-%d)
   instance_name=mdis_demo25 # your_instance_name here
   #local_dump_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/db-dumps
   cd $local_dump_dir;  
   ZIPFILE=mysqldb_backup_172.26.0.17--${current_date}--${instance_name}.sql.zip
   SQLFILE=mysqldb_backup_172.26.0.17--${current_date}--${instance_name}.sql
   unzip -l $ZIPFILE        # have a look at the contents of the zip file
   unzip $ZIPFILE $SQLFILE  # extract the sql file

It is not necessary to unzip the other zip file (fe+be, multi-GB docker-volume) manually. It will be extracted by Vagrant provisioning scripts. You can unzip and inspect the archive for your own information, though. See Vagrantfile, variable zip_file has a hardcoded path to the zip file.

Install VirtualBox and Vagrant on your local machine, if you haven't already.
- Vagrant should be version 2.2 or higher (2.4 ok), installed with the following plugins:
  - vagrant plugin install vagrant-vbguest - no longer developed, but important for shared folders and clipboard sharing between host and guest.
  - vagrant plugin install vagrant-env - read in .env file during provisioning. Passes env vars to guest VM.
  - vagrant plugin install vagrant-disksize - optional, to exert more fine control about the initial size of the virtual hard disk that virtual machines will receive from the host.
  - If any Ruby scripts that entail these plugins contain any calls to the File.exists? method, you'll get deprecation errors undefined method "exists?". Replace File.exists? with File.exist? in those Ruby scripts.
- Vagrant local images**
  - Vagrant installs cached boxes and plugins in $HOME/.vagrant.d/.
  - To list the contants of your local Vagrant box cache, run
  - vagrant box list .
    This will show you a list of all the boxes you have installed locally:
```
   ....
   ubuntu/lunar64                 (virtualbox, 20231129.0.0)
   ubuntu/noble64                 (virtualbox, 20250115.0.1, (amd64))
   .... more boxes ....
```
  - Vagrant will create another .vagrant.d/ directory: in the directory where your Vagrantfile is located. For example, there are ssh keys (file $HOME/.vagrant.d/insecure_private_key used for the very first command vagrant ssh. This key will get replaced with a new one put in <vagrantfile_dir>/.vagrant.d/). See step 6.
Prepare a Linux Image for the guest VM.
To prepare a Linux Image for the guest VM, you have two options, (A) and (B)
(A). Manual creation from scratch. You convert a "safe" Image (e.g. in .ova format) into a Vagrant-compatible box, allowing you to use it locally. (In 2025, we use ubuntu/noble64.) You download that image from trusted site yourself, e.g. from cloud-images.ubuntu.com, add mDIS prerequisites and the mDIS software later.
Details are explained in vagrantbox-from-cloudimage.
(B). Vagrant can download an appropriate image from Hashicorp Cloud. (formerly Vagrantup.com). You do not need an account to download images. Download starts with the command vagrant up.
- Image ubuntu/lunar64 variant from 2023 was an good choice for mDIS instances (Image came with PHP 8.1), but it has been removed from VagrantHub, and it is not recommended to use it anymore.
- As-is, most images on vagrantup.com are not useful or insufficient for mDIS. Often they do not contain a graphical desktop, for example. We install missing 3rd party software later via apt, npm, composer and git. (Gnome desktop will take quite some time to download and install.)
Create a new Vagrantfile for the VirtualBox instance.
Or adapt an existing Vagrantfile via copy-paste. See vagrantfile in any subdirectory at GFZ gitlab server Update the paths to the dump file and the zip file.
- Create an .env file with MYSQL credentials and any files that are needed for the mDIS instance. Alternatively, extract it from the Frontend-Backend (fe-be) zip file, like this: unzip mdis-demo25-fe-be--20240521-rz-vm412.zip .env,
  then copy it to the directory where the Vagrantfile is located.
- Check the variables defined in the Vagrantfile. Import it with the vagrant-env plugin, and/or source it with set -a; source .env; set +a in the terminal. Check the .env file for correct settings, also check shell vars with env | grep ....
- If you have created the Vagrantfile by copying and pasting an entire directory: Delete any copied .vagrant subdirectory from your current/new directory. It contains state information about an existing, different Vagrant Box. This will interfere with the new VM you are about to create.
Adapt further configuration files, e.g., the .env-... files and other ".dotfiles". You might want to change the hostname of the mDIS MySQL/MariaDB database. This is done in the .env file. The .env file could look like this:

MYSQL_PASSWORD=xxx
MYSQL_USER=dis_db_user
MYSQL_HOST=db
MYSQL_DATABASE=mdis_demo25
MYSQL_ROOT_PASSWORD=yyy
MYSQL_DUMPFILE=/vagrant/db-dumps/mysqldb_backup_172.26.0.3--2024-05-21--mdis_demo25.sql
PHP_VERSION=8.1
MYSQL_VERSION=8.0

These are essential variables that are used in the provisioning scripts. (There may be more, e.g., if you install on a Windows host)

Make sure no other VirtualBox VMs are running. At least ensure there will be not conflicts with forwarded SSH ports.
Disconnect from any VPN client. Especially Cisco Secure VPN does not play nicely with private Docker networks or Vagrant Networks. Cisco VPN interferes with the network settings of your VMs. (Edit the routing tables if you can do that, or disconnect from the VPN.)
Check that any of the non-SSH ports 8888, 9003 (or those port numbers you chose for forwarding ssh, webserver, xdebug) are not already in use by other applications. These are used by VirtualBox to forward Debug and HTTP traffic to the VM.
A command for checking the ports in use is sudo lsof -nP -iTCP -sTCP:LISTEN.

Executing the Vagrantfile

Preparatory steps are done. Now actually create the VirtualBox VM.

Execute the Vagrantfile with vagrant up.

This step will create the new VirtualBox VM.ToDo: explain the steps in the Vagrantfile.
If the system hangs during the first boot, with the message:
Guest Screen ... were not restarted automatically,
then there's a problem with automatically installing the VirtualBox Guest extensions. When the installation appears to hang: Open a second terminal on the host (with the settings of the hanging terminal), type vagrant ssh, then after successfully logging into the VM, execute
sudo rcvboxadd reload.
This will reload the VirtualBox Guest additions and might help to get the shared folders working. Then the startup process will continue.