mDIS Backups

System Administration
MySQL Administration

Backup/Restore

Tips

All these sections are very specific to ICDP internal mDIS installations.

Basics

Daily cron jobs

Often there exists a backup job that is scheduled to run automatically on a daily basis.
Check with crontab -l.
The job puts database dumps and form designs into a single zip file. Usually, these files are stored in directories with hardcoded directory names. The dump/backup script is in the $HOME/bin directory.

For the daily dump files, look into

/var/tmp/mysql or
backend/data/upload/backup/mysql

The zip files contain a full database backup and all json files of form-/model definitions.
A typical filename of such a zip file would be
mysqldb_backup_mdis-jet--10.132.0.3--2020-04-03--dis.sql.zip.
For more info about this zip file, see below.

ICDP internal

Full and Incremental backups

Backups can be "full," "differential," or "incremental."

Full backups are copies of the entire system as-is.

The difference in incremental vs. differential backup is that, while an incremental backup only includes the data that has changed since the previous backup, a differential backup contains all of the data that has changed since the last full backup.

Backups from data.icdp-online.org

This image illustrates how backups of ICDP internal mDIS installations are done. Specifically, the mDIS instances on data.icdp-online.org.

They cannot be directly backed up by GFZ Backup infrastructure because the mDIS instances run in an isolated subnet (a DeMilitarized Zone, DMZ) and thus cannot be accessed directly.

This is shown by the red dashed line in the figure below.
Backups are made via an intermediate server, wb45, which is not in the DMZ but has access to mDIS files and directories in the DMZ.

mDIS Backups are made via an intermediate host, wb45 (in 2022)

MySQL database dumps are made once daily as full backups (started via a cron job 1x daily, saved via rsync).
Uploaded files and directories are backed up via rsync. This is done via a cron job (also periodically run, 1x daily).
(Green lines indicate the flow of data and shell commands.)

This way, only changed files (and new files) are copied from the DMZ to the intermediate server. However, the GFZ archive server makes differential backups. This way, retrieving previous versions of files ("going back in time") is still possible.

Still, any "restore backup" workflow is somewhat complicated. Older (snapshotted) files must be retrieved from GFZ Archive to wb45 first and then copied back to the DMZ. (I hope this is clear)

VirtualBox Backups

There exist alternative methods to make backups of science data files and of the whole mDIS installation itself.

Single multi-gigabyte .ova file

A simple way to make a "full" backup is simply to create an .ova image of the virtual machine. An .ova image is a single large multi-gigabyte file that contains everything. The image creation process was described in "Installation with VirtualBox".

You can restore a backup using the same mechanism as installing a virtual server using the *.ova image (see above).
However, this method can consume a lot of disk space in long-running projects, and the act of backing up and/or restoring is quite slow (~10-30 minutes).

Command line script

A simple command line script for making a backup from the command line.

Code sample / Shell script

# get list of VirtualBox images installed on machine

vboxmanage list vms

# pick one VBox from the list,
# then export it to .ova file

vboxmanage export mdis-jet_20200329 -o /data/virtualbox/shared_boxes/mdis-jet_20200329.ova

On Windows, the command would be very similar, except the pathnames would look different, of course.

VirtualBox Snapshots

On the host system, create "Snapshots" with the VirtualBox software. This creates much smaller files than full backups, but they are a bit harder to maintain and restore.
In the list of snapshots, right-click on any snapshot you have taken and select Restore. By restoring a snapshot, you go back or forward in time. The current state of the machine is lost, and the machine is restored to the exact state it was in when the snapshot was taken.
More information on snapshots can be found in the VirtualBox documentation

Additional techniques

The techniques described here both apply to virtual machines and in particular to mDIS instances running natively on a server.

Running a "native" mDIS means running an mDIS instance without VirtualBox. Backing up such instances cannot be accomplished with .ova files or snapshots. Instead:

Use backup software to create backups at the operating system level. This process works by finding all the changed files and saving only these files to backup storage.
- For Ubuntu, this could be Deja Dup, a software to automatically create incremental backups and full backups. See the documentation for that program.
  Install it with
  sudo apt install deja-dup duplicity python-gi.
  The command line program deja-dup also has a GUI frontend which is simply called "Backups." Start it from the command line with deja-dup --name=backups. The configuration via GUI is easy and pretty self-explanatory:

deja-dup-GUI

Additionally, you should have the /var/www/dis directory under version control (git). This protects frontend code and backend code even better if you follow established practices (commit often, push it somewhere).
To save only the database content, you should create a daily dump of the MySQL database (using mysqldump). That will create one single backup file per day, which can be saved away with the chosen backup software (or stored directly in the appropriate location).
See also the section MySQL Administration and mysql-backup-and-restore page. Backing up the MySQL server independently via DB dumps has additional benefits. For example, it enables you to restore individual tables if necessary.

Saving backups

You must also save the backups to offsite storage. That means: other network locations or external media (e.g., USB sticks which are removed and stored elsewhere).

Use git push to save your version-controlled directories to a remote git repo. This protects against scenarios such as failed hardware, stolen laptops, etc.

Restore Operations

This section describes how to restore a corrupt mDIS from a backup.

For each of the above-mentioned backup strategies, there are several ways to do the inverse process, to restore a damaged system.

TBC

Explaining this in appropriate detail would take a lot of space even for the simplest backup method.

Restoring from git

Some hints:

You should have the /var/www/dis directory under version control. If you have several recent branches in your code tree, you can restore a damaged file by copying it from a different branch to the current branch. Do this with
git checkout mybranch ./myfile.
If you have messed things up in a big way, you can then try a
git reset --hard <COMMIT-ID>
to go back in time to a previous state.
Git commands work for frontend and backend code, but not for the database tier and the MySQL data. Thus, for a shell script that can import MySQL backups/dumps, see also the mysql-backup-and-restore page.
Restoring VirtualBox snapshots: see above
TBC

Restoring Database Tables

Try to find a recent backup file that the daily backup cron job creates every day.

Perhaps it resides in /var/tmp/upload/backup/mysql/ and could be named similar to mysqldb_backup_vbox_mdis-jet--10.0.2.15--2020-04-10--dis.sql.zip.

Your system administrator might have configured a different output directory and filename convention, though.

The contents of the file should look like this:

Output of unzip -l /var/tmp/upload/backup/mysql/mysqldb_*--2020-04-10*.zip | grep -v "txt\| 0"

  Length      Date    Time    Name
---------  ---------- -----   ----
## note the first file here - the dump file of the database:
   233326  2020-04-10 10:10   mysqldb_backup_vbox_mdis-jet--10.0.2.15--2020-04-10--dis.sql
## form-definition files, and table definitions:
     2132  2020-04-09 21:27   backend/dis_templates/forms/program.json
     6283  2020-04-09 21:27   backend/dis_templates/forms/expedition.json
     5976  2020-04-09 21:27   backend/dis_templates/forms/site.json
    13192  2020-04-09 21:27   backend/dis_templates/forms/hole.json
    18377  2020-04-09 21:27   backend/dis_templates/forms/core.json
     9719  2020-04-09 21:27   backend/dis_templates/forms/section.json
    11350  2020-04-09 21:27   backend/dis_templates/models/ArchiveFile.json
     7665  2020-04-09 21:27   backend/dis_templates/models/ProjectSite.json
    14298  2020-04-09 21:27   backend/dis_templates/models/ProjectHole.json
    18732  2020-04-09 21:27   backend/dis_templates/models/CoreCore.json
     7880  2020-04-09 21:27   backend/dis_templates/models/CoreSection.json
     7939  2020-04-09 21:27   backend/dis_templates/models/ProjectExpedition.json
     2580  2020-04-09 21:27   backend/dis_templates/models/ProjectProgram.json
---------                     -------
   360345                     19 files

(Shortened)

If your zip file also contains these *.json files, you can also restore mDIS Forms and Models.

Restoring Forms and Models

In order to restore forms and models, you must do this:

For forms, put the respective *.json file into backend/dis_template/forms.

For models/tables, put the respective *.json file into backend/dis_template/templates.

The Code Generator of the Template manager will pick up the uploaded JSON file and generate a new form or model for you.

If the table has not been created yet, start with the .json for the table, then upload the *.json for the form.

Replacing Forms is easy, but Replacing Tables is harder

It is easy to restore a data-entry form by uploading JSON files. The form will be created instantly because a form is a standalone object in mDIS.

It is harder to restore tables. This has to be done in a certain sequential order, and even if the restoring operation succeeds, the table will be empty. Loading the newly restored table with science data (that conforms to the new schema of the restored table) is another matter and can be quite a tedious task.

Restoring MySQL Objects

Filling empty MySQL tables must be done in a certain sequential order and requires working with the mDIS File-Importer, which is part of the "File-Upload" GUI feature, but see also the developer docs.

For low-level access, try the Adminer MySQL Administration tool (if it is installed with mDIS).

For details describing the process of restoring the mDIS MySQL database, also read this extra document.

Restoring VirtualBox Instances

Full .ova file: Simply load the .ova file into VirtualBox. You can use the interactive tool "VirtualBox Manager" to restore to a new virtual machine or to overwrite an older one.

Restore a VirtualBox snapshot:

  # vboxmanage snapshot <VM-NAME> restore <SNAPSHOT-NAME>
  # TBC
  # rarely done this myself