CI/CD and Build System
CI/CD and Build System
ICDP internal: This section describes the mDIS build pipeline, runners, and GitLab CI/CD settings.
Start Here
Regular Users: read the mDIS User Documentation
Developers: who need to build and deploy mDIS instances (online webapps), read on.
- Read
README.mdof the mDIS Source Code Repository first. However, that README.md contains only an outline.
These pages contain additional information about somewhat automated installations. - These pages are work in progress, since 2023
- Read
The mDIS Build and Installation System
Our development workflow with automated build steps:
flowchart LR
Developers -->|"Code Push"| VersionControl["Version Control<br>(GitLab)"]
VersionControl -->|"triggers CI"| CIPlatform["Build System<br>(GitLab)"]
CIPlatform --> DockerImage["mDIS Docker image<br>(GitLab)"]
CIPlatform -- "delivers Feedback " --> Developers
CI = Continuous Integration
In contrast, building mDIS instances manually can be time-consuming and error-prone.
Especially,
- if we need to repeat these build processes many times.
- if we need to create many variants of installations for testing and comparison purposes.
- we may need to restart or reconfigure these installation workflows, after long breaks.
- or if these tasks need to be carried out by different people, with different levels of expertise.
Therefore, we need something to automate, monitor, and govern the mDIS build and deployment processes.
For Experts
- mDIS Build System: Step-by-Step Guide - General - ICDP Internal.
- mDIS Build System: Step-by-Step Guide - Detailed - ICDP Internal.
- WiP notes
For Non-Experts
Build System: CI/CD Concepts
Non-Experts who need to understand the basics of CI/CD, start here:
Visual Overview Diagrams:
- Goals and Characteristics of the mDIS Build System
- Runners and Pipelines basics
- Build Pipeline Flowchart (long, detailed diagram)
- mDIS Build Workflows: - flowchart diagrams (BPMN standard)
- mDIS General Deployment scheme - reverse proxy concept
More Information about this wiki.
Build System: CI/CD Settings
It a major benefit to be able to reconfigure the CI/CD system. Thus we can start various automated workflows, in many ways.
Important configuration variables
- CI/CD Settings: Gitlab Variables - Credentials, base-URLs, and other settings (project-specific Path fragments, etc.)
- CI/CD Settings: Gitlab Runners - we use several runners (software that executes the CI/CD jobs on dedicated machines)
- Reports and Artifacts
- For details, see mDIS Build Reports and Artifacts (extra page)
ICDP Internal
Internal Documents
These links describe the Build System of the mDIS Source Code, in particular
- Step-by-Step Guide - ICDP Internal
- the Gitlab CI/CD Settings of the
master--git-gfz--ci-cdbranch, and related branchesgit-...YYYYMMDD-COMMITHASH,foundation-imagesundblank-images.. - its
.gitlab-ci.ymlfiles and
Target audience of these pages are mDIS Project members - ICDP data management, GFZ developers, GFZ employees, and external collaborators.
Misc
- optional: (from the mDIS Documentation): Manual Installation with Docker - extremely verbose, but for Docker and Docker-compose only. Background information.
- optional: read Installation with VirtualBox - very verbose, but for VirtualBox only.
- optional: read How to push a running mDIS Docker Image to GFZ's container registry
- More - Scratchpad, How to contribute to this wiki...
- ToDo items - unresolved CI/CD issues, and other notes.
- Contribution Guidelines for this wiki.
- mDIS-installer
- Vagrant Provisioning Scripts for converting a backup of an online instanec to an offline instance
- Shellscripts