Monitoring servers - ICDP internal

(Continuation from sys-admin-docker.md)

Work In Progress

Just a bunch of Various shell commands and notes for monitoring servers. (Perhaps someday for troubleshooting, too.)

These commands use a simple approach to monitor the running state of a container on a remote server:

Log in to the server and run the commands there.

All running mDIS containers can be checked with their friendly container name, e.g. mdis_php_grind_1:

On wb45 run:

sudo -u Nagios /etc/nagios3/conf.d/check_docker_by_ssh -c mdis_php_grind_1

If this doesn't work, then run on rz-vm412:

$HOME/bin/docker_nagios/dkc_status.sh -c mdis_php_grind_1

Expected output:

OK: mdis_php_grind_1 status is running

Based on script (produces slightly more output): $HOME/bin/docker_nagios/check_docker --connection /var/run/docker.sock --cpu 10:20 --timeout 4 2>/dev/null

Connect to Docker Daemon via encryption, TCP socket

Administering Docker containers remotely has some key advantages. For example, you can check the status of a container, restart it, or even stop it, without having to log in to the server. You can also monitor the container's resource usage, such as CPU and memory.

However, communications between Docker Clients and the dockerd Service are insecure by default. Besides, the dockerd runs with root privileges. So it is essential to restrict and encrypt the connections that can access dockerd remotely.

Install a Certificate Authority (CA) on the Docker Host, and generate a Server Certificate and a Key for the Docker Daemon. Also generate a Client Certificate and a Key for each Docker client that will connect to the Docker Daemon via the encrypted TCP socket.

Certificates and 3rd-party Certificate Chain files can be created with the openssl and mkcert commands. The mkcert command is a wrapper around openssl that simplifies the process.

TBA 😃

After Cert/Key Generation

Edit /etc/docker/daemon.json:

...TBC...

Restart the Docker Daemon.

Check if TCP socket connection is enabled for Docker daemon.

Port 2376 must be open. This port is the standard port for an encrypted TCP socket connection. For security reasons, it should only accept connections from localhost or from hosts that you trust.

Check if the port is open:

# as root on localhost
nmap --open localhost

# or more specifically
sudo nmap -PN -sT -p 2376 139.17.229.12

PORT     STATE SERVICE
2376/tcp open  Docker

Check if the firewall allows connections to this port:

sudo ufw status numbered

This command may return a firewall rule that allows access to this port from the Nagios host.

[7] 2376 ALLOW IN 139.17.<...> ### GFZ's Class B Network

TLS/SSL

For encrypted communications, the certificates and keys must be stored in a directory. Alternatives are:

/etc/ssl/certs/       # Debian/Ubuntu standard dir
/usr/share/ca-certificates/       # for your own certificates
/usr/local/share/ca-certificates/ # and chain files

TBC