mDIS User Documentation

Converting an online mDIS instance to VirtualBox instance

WORK IN PROGRESS

see also: Installation with VirtualBox - advanced topics

In this article, "Online instance" means "mDIS running as a Docker container on a remote server".

Why convert an online instance to a VirtualBox instance?

This might become necessary

  • if you want to use the instance somewhere where there is no internet- doing fieldwork in remote locations, perhaps.
  • Another scenario is that you want to make changes to your own local copy of the instance without affecting the online version.
  • Another scenario is you want to reduce IT security measures for a local copy of the instance, e.g., to allow for more convenience in development and for everyday users

Note

These instructions are ICDP-OSG-specific. They assume that you have access to the online instance and that you have the necessary permissions to access all parts of its infrastructure (e.g., the network). They also mention some very specific details such as certain hostnames and paths. You will need to adapt these instructions to your own situation.

At a glance

Strategy

  1. Find a recent backup of the online instance
  2. Download the backups to your local machine.
  3. Install VirtualBox and Vagrant on your local machine, if you haven't already. Read this guide
  4. Decide on a Linux Image for the guest VM. Vagrant will download that image autonomously from Vagranthub (opens new window) in step 8.
  5. Find the Vagrant provisioning scripts in charge of installation subtasks (file-copying, and shellscript-executions).
  6. Create a new Vagrantfile for the VirtualBox instance. Or adapt an existing Vagrantfile via copy-paste.
  7. Adapt further configuration files, e.g., the .env-... files and other ".dotfiles".
  8. Execute the Vagrantfile with vagrant up.
  9. Do finetuning of the Vagrantfile and the provisioning scripts. Loop between steps 6-9 until both the guest and host run smoothly.
  10. Do finetuning of the mDIS instance itself. Work with mDIS forms as you usually would.
  11. Export the VirtualBox instance to a .ova file, if you want to share it with others.

Step-by-step instructions

These instructions show in greater detail how to convert an online mDIS instance to a VirtualBox instance.

  1. Find a recent backup of the online instance
    • Alternatively, create a new backup of the online instance.
    • Download the backup to your local machine.
      • The backup is a .zip file for the sql database dump
      • For the PHP-based backend code, you can copy the entire docker volume to your local machine.
  2. Download the backups to your local machine.
    • Command: scp -r user@remote:/path/to/backup /path/to/local/backup
    • For wb33 PC:
      • fetch mDIS Database dump:
   dump_zip_dir=/data2/backup/rz-vm412/mdis-sqldumps
   dump_zip_file=mysqldb_backup_172.26.0.3--2024-05-02--mdis_namcore.sql.zip
   local_dump_dir=$HOME/code/git/_my/work/mdis-installer/shared-common/db-dumps
   scp -r wb45:$dump_zip_dir/$dump_zip_file $local_dump_dir; cd $local_dump_dir
1
2
3
4
  • fetch mDIS docker volume:
     volume_dir=/data1/docker/docker-data/volumes/mdis_namcore_web/_data  
     ssh rz-vm412.gfz-potsdam.de "cd $volume_dir && zip -r - ." > mdis-namcore-fe-be--20240502-rz-vm412.zip
1
2
  • This keeps a local copy of the current state of the online instance. 2 Files are downloaded:
    • mysqldb_backup_172.26.0.3--2024-05-02--mdis_namcore.sql.zip (a few MB in size)
    • mdis-namcore-fe-be--20240502-rz-vm412.zip ~x00 MB compressed for upcoming projects, but many GBs for long-running projects. Even larger in size when uncompressed. Check your disk spaces before downloading and decompressing.
    • Optional: You can unzip the big fe-be dump locally after transfer, remove directories that you do not need, and re-zip it. This will save space on your local machine. If you re-zip, do not forget to zip dotfiles and hidden directories as well.
    • files in directories backend/vendor and frontend/node_modules containing 3rd-party dependencies are also included in the backup. They are needed for the VirtualBox instance, but some of them might need to get recompiled (.c files in node_modules/ directory). This depends on your VM image and the version of the dependencies you require.
  1. Unzip the database sql file on your local machine
    • Commands:
   cd $local_dump_dir;  
   ZIPFILE=mysqldb_backup_172.26.0.3--2024-05-02--mdis_namcore.sql.zip
   SQLFILE=mysqldb_backup_172.26.0.3--2024-05-02--mdis_namcore.sql
   unzip -l $ZIPFILE        # have a look at the contents of the zip file
   unzip $ZIPFILE $SQLFILE  # extract the sql file
1
2
3
4
5

The other zipfile (fe+be, multi-GB docker-volume) will be extracted by vagrant provisioning scripts. See Vagrantfile , variable zip_file has a hardcoded path to the zipfile. ``.

  1. Install VirtualBox and Vagrant on your local machine, if you haven't already.
    • vagrant should be version 2.2 or higher (2.4 ok), installed with the following plugins:
      • vagrant plugin install vagrant-vbguest - no longer developed, but important for shared folders and clipboard sharing between host and guest.
      • vagrant plugin install vagrant-env - read in .env file during provisioning and pass to guest VM.
      • vagrant plugin install vagrant-disksize - optional, to exert more fine control about initial size of virtual harddisk that virtual machines will receive from host.
    • Vagrant installs stuff in $HOME/.vagrant.d/ and in /home/knb/.vagrant.d/boxes/box-image-name>/<box-version-string>/virtualbox/ there are ssh keys (insecure_private_key used for command vagrant ssh), boxes, and plugins.
  2. Decide on a Linux Image for the guest VM, Vagrant will download it from Vagrantup.com.
    • An ubuntu/lunar64 (opens new window) variant from 2023 was an ok choice for mDIS instances (Image came with PHP 8.1).
    • Image ubuntu/vagrant64 was available in 2024. Came with PHP 8.2.
    • Most images on vagrantup.com do not contain a graphical desktop, though. We install that later via apt. (Gnome desktop will take quite some time to download and install.)
  3. Create a new Vagrantfile for the VirtualBox instance.
    Or adapt an existing Vagrantfile via copy-paste. Update the paths to the dumpfile and the zipfile.

    • create an .env file with MYSQL credentials and any files that are needed for the mDIS instance. Recommended: extract it from the fe-be zipfile, like this: unzip mdis-namcore-fe-be--20240521-rz-vm412.zip .env,
      then copy it to the directory where the Vagrantfile is located. Import it with vagrant-env plugin, and/or source it with set -a; source .env; set +a in the terminal.

    • check the variables defined in the Vagrantfile.

  4. Adapt further configuration files, e.g., the .env-... files and other ".dotfiles". You might want to change the hostname of the mDIS MySQL/MariaDB database. This is done in the .env file. The .env file could look like this:
MYSQL_PASSWORD=xxx
MYSQL_USER=dis_db_user
MYSQL_HOST=mdis_db_1
MYSQL_DATABASE=mdis_namcore
MYSQL_ROOT_PASSWORD=yyy
MYSQL_DUMPFILE=/vagrant/db-dumps/mysqldb_backup_172.26.0.3--2024-05-21--mdis_namcore.sql
PHP_VERSION=8.1
MYSQL_VERSION=8.0
1
2
3
4
5
6
7
8

These are variables that are used in the provisioning scripts. (There may be more, e.g.if you install on a Windows host) 5. Make sure no other VirtualBox VMs are running. There may be conflicts with forwarded ports. 6. Disconnect from any VPN client. Especially Cisco Secure VPN does not play nicely with private Docker networks, or Vagrant Networks. Cisco VPN interferes with the network settings of your VMs. 7. Check that any of the ports 2222, 8888, 9003 are not already in use by other applications. These are used by Vagrant to forward SSH and HTTP traffic to the VM. 8. Execute the Vagrantfile with vagrant up. ToDo: explain the steps in the Vagrantfile. -. If the system hangs during first boot( building the VirtualBox Guest extensions), open a second temrinal, vagrant ssh, then inside the VM execute sudo rcvboxadd reload. This will reload the VirtualBox Guest additions and might help to get the shared folders working. The boot process will continue. 9. Do finetuning of the Vagrantfile and the provisioning scripts called from within the Vagrantfile. - to execute the provisioning scripts individually, you can use the command
vagrant provision --provision-with mdis-db-import to execute only the mdis-db-import . provisioner/script (in case that step failed during the initial vagrant up). However,, then the /vagrant dir will be unavailable after the script has run. 10. Do finetuning of the mDIS instance itself. - check apache config files, e.g.,
/etc/apache2/sites-available/000-default.conf - DocumentRoot
/etc/apache2/conf-available/globalname.conf - ServerName
Check if .htaccess files are in place (in dis/web dir), and if they allowed to override settings (check AllowOverride in /etc/apache2/apache2.conf) - run env|grep MYSQL, check that the credentials are set. Check .env file for correct settings. If there is no .env file, create one by copying the ..env-example file or by copying the .env file you used for the vagrant up command. To import them on the command line, run set -a; source .env; set +a
Check if database is reachable with php icdp_work/php-check-db-connectivity.php. Should return MariaDB/mysql version.
Check if ./yii config command gives meaningful output with these settings. It should not crash.
If you ran npm install --clean-install in the frontend directory, you might need to run npm run build to recompile the frontend code.
If you deleted the backend/vendor dir of 3rd party code dependencies, run composer install to get them back from the internet, and "recompile" the backend code.
Check if the backend/runtime directory is writable by the webserver.
Check if the apache webserver runs as the vagrant user. Check if the vagrant user is in the sudoers group (simply execute a sudo command).
Check scripts in the /home/vagrant/bin/ directoy and re-run them if necessary. (Some perform actions, some just list information.)
Set the password for the ubuntu user: sudo passwd ubuntu. Ignore BAD PASSWORD message. Check the backup-cronjob in crontab -l and run it manually at least once if necessary. Fix any errors (check for 'No such file or directory' errors, create missing subdirs if necessary...).
Check the recency of the gitlab repo in /var/www/dis with git status. Does it refelct the state of the repo on the "online" server? (ToDo: Add more checks here.) 11. If you installed a Graphical Desktop, - you need to login as Linuxuser "vagrant" at least once. And start your favorite applications at least once. This will get rid of welcome screens and other first-time-run dialogs. - If you installed Gnome, you can change the session type from Wayland to X11 in the login screen. This might help with clipboard sharing between host and guest. - If still no clipboard, re-install the guest additions by: Adding a SATA controller to the VM, and attaching the VBoxGuestAdditions.iso to it. Then check where the DVD was mounted, then, inside the guest run sudo /media/vagrant/<mountdirectory>/VBoxLinuxAdditions.run. Reboot the VM.
- Optional: Check the sidebar ("dock") of the GNOME desktop, pin additional favorite apps. 12. Export the VirtualBox instance to an .ova file, if you want to share it with others.
- Command: vboxmanage export $vm -o $outfile
with $vm being the name of the VM obtained with vboxmanage list vms and $outfile being something like mdis-namcore-20240502.ova.
Optional: Inspect the .ova file with tar -tvf and tar -xvf to see what is inside. 13. Import the .ova file into VirtualBox on another machine with vboxmanage import $outfile, or use the Graphical "Import appliance" dialog in VirtualBox Manager. Change shared folder settings to your situation.

See also

Installation with VirtualBox - advanced topics