Converting an online mDIS instance to VirtualBox instance

WORK IN PROGRESS

See also: Installation with VirtualBox - advanced topics

In this article, "Online instance" means "mDIS running as a Docker container on a remote server".

Why convert an online instance to a VirtualBox instance?

This might become necessary:

• If you want to use the instance somewhere where there is no internet, such as during fieldwork in remote locations.
• Another scenario is that you want to make changes to your own local copy of the instance without affecting the online version.
• Another scenario is you want to reduce IT security measures for a local copy of the instance, e.g., to allow for more convenience in development and for everyday users.

Note

These instructions are ICDP-OSG-specific. They assume that you have access to the online instance and that you have the necessary permissions to access all parts of its infrastructure (e.g., the network). They also mention some very specific details such as certain hostnames and paths. You will need to adapt these instructions to your own situation.

Step-by-step instructions

These instructions show in greater detail how to convert an online mDIS instance to a VirtualBox instance.

Steps 1-10 are preparatory steps.

1. Find a recent backup of the online instance.
   • Alternatively, create a new backup of the online instance.
   • Download the backup to your local machine.
     • The backup is a .zip file for the SQL database dump.
     • For the PHP-based backend code, you can copy the entire Docker volume into a zipfile on your local machine. This creates a most recent copy of the entire PHP backend code.
2. Download the backups to your local machine.
   • Command: scp -r user@remote:/path/to/backup /path/to/local/backup
   • For wb33 PC:
     • Fetch mDIS Database dump:

#!/usr/bin/env bash
# This script fetches the most recent backup of the mDIS database from a backup_server
# call as: download-recent-mdis-backup.sh mdis_goedeep
current_date=$(date +%Y-%m-%d -d "yesterday")
instance_name=$1 #"your_instance_name"
backup_server=somehost.gfz.de
filename_prefix=mysqldb_backup_data.icdp-online.org
if [ -z "$instance_name" ]; then
    echo "Please provide the instance name  the first argument, "
    echo "(as it appears in backup files, e.g. mdis_demo25)"
    exit 1
fi
# Copy from server which stores DB backups to local machine
dump_zip_dir=/data2/backup/rz-vm412/mdis-sqldumps
dump_zip_file=${filename_prefix}--${current_date}--${instance_name}.sql.zip
local_dump_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/db-dumps
scp -r $backup_server:$dump_zip_dir/$dump_zip_file $local_dump_dir
echo "now continue here:"
echo "cd $local_dump_dir"

#local_dump_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/db-dumps
cd $local_dump_dir;  
ZIPFILE=${filename_prefix}--${current_date}--${instance_name}.sql.zip
SQLFILE=${filename_prefix}--${current_date}--${instance_name}.sql
unzip -l $ZIPFILE        # have a look at the contents of the zip file
unzip $ZIPFILE $SQLFILE  # extract the sql file

ls -lrt  $local_dump_dir | tail -n 4

• fetch mDIS Docker Volume, which contains the entire /var/www/dis directory.
  (expect to download a few 100 MB for new/upcoming projects, but expect many GBs for long-running projects). Even larger in size when downloading without zip compression, uncompressed. Check your disk space before downloading and decompressing.

#!/usr/bin/env bash
# This script fetches the most current state of an mDIS docker volume from a production server
instance_name=$1 #"your_instance_name"
if [ -z "$instance_name" ]; then
    echo "Please provide the instance name  the first argument, "
    echo "(as it appears in backup files, e.g. mdis_demo25)"
    exit 1
fi
# Copy from server which stores DB backups to local machine
current_date=$(date +%Y%m%d -d "yesterday")
mdis_host=somehost.gfz.de
volume_dir=/data1/docker/docker-data/volumes/${instance_name}_web/_data  

local_zip_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/${instance_name}-${mdis_host}
mkdir -p "$local_zip_dir"
cd "$local_zip_dir"
ssh $mdis_host "cd $volume_dir && zip -r - ." > "${instance_name}"-fe-be-"${current_date}"-"${mdis_host}".zip

• In the above ssh command, the content of the prospective zip file is not saved to a temporary zip file on the remote machine. Instead, it is created and streamed directly over the network using an in-memory buffer. The zip command outputs the zipped content to stdout, which is then piped through the SSH connection to the local machine. This approach avoids the need for temporary storage (of a potentially very large zip file) on the remote machine.
• Summary so far: This created a zipped local copy of the current state of the online instance. 2 Files were downloaded to the local machine:
  • mysqldb_backup_data.icdp-online.org--2024-05-02--mdis_demo25.sql.zip (a few MB in size)
  • mdis-demo25-fe-be-20240502-rz-vm412.zip (here ~500 MB compressed data).
  • Optional: You can unzip the big Frontend-Backend (fe-be) dump locally after transfer, remove some directories that you do not need, and re-zip it. This will save space on your local machine. If you re-zip, do not forget to zip dotfiles and "dot-directories" as well.
  • For example, Files in directories backend/vendor and frontend/node_modules containing 3rd-party dependencies are also included in the backup. You may delete those. Also some of them might need to get recompiled in your new Vagrant box (for example, .c files in the node_modules/ directory might need recompilation). This depends on your VM image and the versions of the dependencies you require. If your machine is similar to the source machine, you might not need to recompile. This is a complex topic not discussed in this guide.

3. Unzip the database SQL file on your local machine
   • Commands:

   # see above,
   # script: download-recent-mdis-backup.sh mdis_goedeep
   current_date=$(date +%Y-%m-%d)
   instance_name=mdis_demo25 # your_instance_name here
   #local_dump_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/db-dumps
   cd $local_dump_dir;  
   ZIPFILE=mysqldb_backup_172.26.0.17--${current_date}--${instance_name}.sql.zip
   SQLFILE=mysqldb_backup_172.26.0.17--${current_date}--${instance_name}.sql
   unzip -l $ZIPFILE        # have a look at the contents of the zip file
   unzip $ZIPFILE $SQLFILE  # extract the sql file

It is not necessary to unzip the other zip file (fe+be, multi-GB docker-volume) manually. It will be extracted by Vagrant provisioning scripts. You can unzip and inspect the archive for your own information, though. See Vagrantfile, variable zip_file has a hardcoded path to the zip file.

4. Install VirtualBox and Vagrant on your local machine, if you haven't already.
   • Vagrant should be version 2.2 or higher (2.4 ok), installed with the following plugins:
     • vagrant plugin install vagrant-vbguest - no longer developed, but important for shared folders and clipboard sharing between host and guest.
     • vagrant plugin install vagrant-env - read in .env file during provisioning. Passes env vars to guest VM.
     • vagrant plugin install vagrant-disksize - optional, to exert more fine control about the initial size of the virtual hard disk that virtual machines will receive from the host.
     • If any Ruby scripts that entail these plugins contain any calls to the File.exists? method, you'll get deprecation errors undefined method "exists?". Replace File.exists? with File.exist? in those Ruby scripts.
   • Vagrant local images**
     • Vagrant installs cached boxes and plugins in $HOME/.vagrant.d/.
     • To list the contants of your local Vagrant box cache, run
     • vagrant box list .
       This will show you a list of all the boxes you have installed locally:

        ....
        ubuntu/lunar64                 (virtualbox, 20231129.0.0)
        ubuntu/noble64                 (virtualbox, 20250115.0.1, (amd64))
        .... more boxes ....
     
     • Vagrant will create another .vagrant.d/ directory: in the directory where your Vagrantfile is located. For example, there are ssh keys (file $HOME/.vagrant.d/insecure_private_key used for the very first command vagrant ssh. This key will get replaced with a new one put in <vagrantfile_dir>/.vagrant.d/). See step 6.
5. Prepare a Linux Image for the guest VM.
   To prepare a Linux Image for the guest VM, you have two options, (A) and (B)
   (A). Manual creation from scratch. You convert a "safe" Image (e.g. in .ova format) into a Vagrant-compatible box, allowing you to use it locally. (In 2025, we use ubuntu/noble64.) You download that image from trusted site yourself, e.g. from cloud-images.ubuntu.com, add mDIS prerequisites and the mDIS software later.
   Details are explained in vagrantbox-from-cloudimage.
   (B). Vagrant can download an appropriate image from Hashicorp Cloud. (formerly Vagrantup.com). You do not need an account to download images. Download starts with the command vagrant up.
   • Image ubuntu/lunar64 variant from 2023 was an good choice for mDIS instances (Image came with PHP 8.1), but it has been removed from VagrantHub, and it is not recommended to use it anymore.
   • As-is, most images on vagrantup.com are not useful or insufficient for mDIS. Often they do not contain a graphical desktop, for example. We install missing 3rd party software later via apt, npm, composer and git. (Gnome desktop will take quite some time to download and install.)
6. Create a new Vagrantfile for the VirtualBox instance.
   Or adapt an existing Vagrantfile via copy-paste. See vagrantfile in any subdirectory at GFZ gitlab server Update the paths to the dump file and the zip file.
   • Create an .env file with MYSQL credentials and any files that are needed for the mDIS instance. Alternatively, extract it from the Frontend-Backend (fe-be) zip file, like this: unzip mdis-demo25-fe-be--20240521-rz-vm412.zip .env,
     then copy it to the directory where the Vagrantfile is located.
   • Check the variables defined in the Vagrantfile. Import it with the vagrant-env plugin, and/or source it with set -a; source .env; set +a in the terminal. Check the .env file for correct settings, also check shell vars with env | grep ....
   • If you have created the Vagrantfile by copying and pasting an entire directory: Delete any copied .vagrant subdirectory from your current/new directory. It contains state information about an existing, different Vagrant Box. This will interfere with the new VM you are about to create.
7. Adapt further configuration files, e.g., the .env-... files and other ".dotfiles". You might want to change the hostname of the mDIS MySQL/MariaDB database. This is done in the .env file. The .env file could look like this:

# Driver name could be the following: 'sqlsrv' for ms sql serve, 'mysql' for mysql server
MYSQL_DRIVER_NAME=mysql
MYSQL_PASSWORD=xyz123
MYSQL_ROOT_PASSWORD=abc987
MYSQL_USER=mdis_demo25
# your backup-zip-dump likely contains a different hostname unlikely to be reachable.
# so change that to localhost
MYSQL_HOST=localhost
MYSQL_DATABASE_WORK=work_something
MYSQL_DATABASE=mdis_demo25
MYSQL_DUMPFILE=/vagrant/db-dumps/name-of-sql-dumpfile.sql
PHP_VERSION=8.3
MYSQL_VERSION=8.0
# alternatively: MYSQL_CRYPT_PASSWORD=.......
# Crypted password can be created using yii config/encrypt-password
#YII_DEBUG=0
#YII_ENV=dev
#PHP_ENABLE_XDEBUG=0
EXTERNAL_ALLOWED_IP=*.*.*.*
# On systems behind a reverse proxy, enter the base url the user sees in his browser
# BASE_URL=

These are essential variables that are used in the provisioning scripts. (There may be more, e.g., if you install on a Windows host)

8. Make sure no other VirtualBox VMs are running. At least ensure there will be not conflicts with forwarded SSH ports.
9. Disconnect from any VPN client. Especially Cisco Secure VPN does not play nicely with private Docker networks or Vagrant Networks. Cisco VPN interferes with the network settings of your VMs. (Edit the routing tables if you can do that, or disconnect from the VPN.)
10. Check that any of the non-SSH ports 8888, 9003 (or those port numbers you chose for forwarding ssh, webserver, xdebug) are not already in use by other applications. These are used by VirtualBox to forward Debug and HTTP traffic to the VM.
    A command for checking the ports in use is sudo lsof -nP -iTCP -sTCP:LISTEN.

Executing the Vagrantfile

Preparatory steps are done. Now actually create the VirtualBox VM.

11. Execute the Vagrantfile with vagrant up.

• This step will create the new VirtualBox VM.ToDo: explain the steps in the Vagrantfile.
• If the system hangs during the first boot, with the message:
  Guest Screen ... were not restarted automatically,
  then there's a problem with automatically installing the VirtualBox Guest extensions. When the installation appears to hang: Open a second terminal on the host (with the settings of the hanging terminal), type vagrant ssh, then after successfully logging into the VM, execute
  sudo rcvboxadd reload.
  This will reload the VirtualBox Guest additions and might help to get the shared folders working. Then the startup process will continue.

12. Do fine-tuning of the Vagrantfile and the provisioning scripts called from within the Vagrantfile.
    • To execute the provisioning scripts individually, you can use the command
      vagrant provision --provision-with mdis-db-import to execute only the mdis-db-import provisioning step (in case that step failed during the initial vagrant up). However, then the /vagrant shared folder may be unavailable after the script has run. In this case you might try the vagrant upload command to upload files to the VM.
      vagrant reload --provision can be run to continue after an installation process was interrupted (e.g., because the zipfile or the sql-dumpfile was not found).
13. Do fine-tuning of the mDIS instance itself.
    • Check Apache config files, e.g.,
      /etc/apache2/sites-available/000-default.conf - DocumentRoot
      /etc/apache2/conf-available/globalname.conf - ServerName
      Check if .htaccess files are in place (in dis/web dir), and if they are allowed to override settings (check AllowOverride in /etc/apache2/apache2.conf)
    • Run env|grep MYSQL, check that the credentials are set. Check the .env file for correct settings. If there is no .env file, create one by copying the ..env-example file or by copying the .env file you used for the vagrant up command. To import .env file contents via the command line, run
      set -a; source .env; set +a .
      Check if the database is reachable with
      php icdp_work/php-check-db-connectivity.php. This should return 1 line, the MariaDB/MySQL version running inside the VirtualBox instance.
      Check if ./yii config command gives meaningful output with the environment variables loaded from .env file. It should not crash. Make sure the MYSQL_HOST is set to localhost. If you ran npm install --clean-install in the frontend directory, you might need to run npm run build to recompile the frontend code. Ignore warnings about deprecated packages.
      If you deleted the backend/vendor dir of 3rd party code dependencies, run composer install to get them back from the internet and "recompile" the backend code.
      Check if the backend/runtime directory is writable by the webserver.
      Check if the Apache webserver runs as the vagrant user. Check if the vagrant user is in the sudoers group (simply execute a sudo command).
      Check scripts in the /home/vagrant/bin/ directory and re-run them if necessary. (Some scripts perform customization actions, some just list information.)
      Set the passwords for the vagrant and ubuntu users: sudo passwd vagrant, sudo passwd ubuntu. Ignore BAD PASSWORD message. If user ubuntu does not exist, do not create it.
      Check the backup-cron job in crontab -l and run it manually at least once if necessary. Fix any errors (check for 'No such file or directory' errors, create missing subdirs if necessary...).
      Check the recency of the GitLab repo in /var/www/dis with git status. Does it reflect the state of the repo on the "online" server? (ToDo: Add more checks here.)
14. If you installed a Graphical Desktop,
    • you need to log in as Linux user "vagrant" at least once. And start your favorite applications at least once. This will get rid of welcome screens and other first-time-run dialogs.
    • If you installed Gnome, you can change the session type from Wayland to X11, and back from X11 to Wayland in the Login screen. This might help with clipboard sharing between host and guest.
    • If still no clipboard, re-install the Guest Additions by: Adding a SATA controller to the VM and attaching the VBoxGuestAdditions.iso to it. Then check where the DVD was mounted, then, inside the guest run sudo /media/vagrant/<mountdirectory>/VBoxLinuxAdditions.run. Reboot the VM.
    • Optional: Check the sidebar ("dock") of the GNOME desktop, pin additional favorite apps.
15. Check if the shared Clipboard is working. Re-install the VirtualBox Guest Extensions. Switch Between wayland and X11 session in the login screen, login to desktop. Adjust the display resolution in the Guest VM settings. Log out again. Disable and enable shared clipboard.
16. Export the VirtualBox instance to an .ova file if you want to share it with others.
    • Command: vboxmanage export $vm -o $outfile
      with $vm being the name of the VM obtained with vboxmanage list vms and $outfile being something like mdis-demo25-20240502.ova.
      Optional: Inspect the .ova file with tar -tvf and tar -xvf to see what is inside.
17. On another machine, import the .ova file into VirtualBox with vboxmanage import $outfile, or use the Graphical "Import appliance" dialog in VirtualBox Manager. Change shared folder settings to your situation.
18. If you once ran an older version of the same mDIS instance on the same host and port before: clear the browser Local Storage (in browser dev tools, run JS command localStorage.clear()), otherwise some model definitions might not work as expected.
19. Try to connect to the VM via SSH, create a shortcut in you ~/.ssh/config. Mine looks like this:

Host vagrantvm
  Hostname localhost
  User vagrant
  Port 2222
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  IdentityFile ...vagrant_dir/.../insecure_private_key
  # IdentityFile ~/.vagrant.d/id_rsa_vagrant_not_insecure

On host, copy ~/.vagrant.d/id_rsa_vagrant_not_insecure.pub to the VM, add to
~/.ssh/authorized_keys file. Enter the vm with ssh vagrantvm. There will be warnings about the host key now logging in to a different host than last time. This is expected, as the VM is a new one. 20. Try to connect to the mysql server running inside the VM. I had to explicitly create a new mysql user privilege with:

create user 'root'@'_gateway';
SET PASSWORD FOR 'root'@'_gateway' = 'secret/root'
GRANT ALL PRIVILEGES ON *.* TO 'root'@'_gateway' WITH GRANT OPTION;
FLUSH PRIVILEGES;
# same for other mysql users

Then you can connect to the mysql server from the host with:

mysql -h 127.0.0.1  -P 33306 -u mdis_demo25 -pSecret/userpassword
# -h localhost is not working for some reason. -P 33306 is my forwarded port

The standard CREATE USER mysql command syntax someone@% does not work (on my Linux).

See also

Installation with VirtualBox - advanced topics