fixes of setup script, creating better template files, ensure coorect latency output to influx

.gitignore

@@ -1,3 +1,7 @@
 .DS_Store
 .vscode
 .env
+*.backup
+grafana-datasource.yml
+docker-compose.yml
+config-template.ini

Docker_Build/latency_monitor.py

@@ -67,7 +67,8 @@ class MyInfluxDB():
         # create influxdb client
         self.client = InfluxDBClient(url=INFX_URL,
                                      token=INFX_TOKEN,
-                                     org=INFX_ORG)
+                                     org=INFX_ORG,
+                                     verify_ssl=False)
 
         # create influxdb write api
         self.write_api = self.client.write_api(write_options=WriteOptions(batch_size=INFX_BATCH,
@@ -81,7 +82,7 @@ class MyInfluxDB():
     def write(self, host, host_location, ping_response):
         self.host = host
         self.host_location = host_location
-        self.ping_response = ping_response
+        self.ping_response = float(ping_response)
         self.influx_timestamp = int(time_ns())
         self.data_point = Point("latency_monitor").tag("location", self.host_location).tag("host", self.host).field("latency", self.ping_response).time(self.influx_timestamp)
         self.write_api.write(bucket=self.INFX_BUCKET,
@@ -113,7 +114,7 @@ class ThreadPing(Thread):
         self.starttime = time()
         while True:
             self.ping_response_list = ping(self.host, count=1)
-            self.ping_response = self.ping_response_list.rtt_avg_ms
+            self.ping_response = "{:.2f}".format(self.ping_response_list.rtt_avg_ms)
             self.MyDB.write(self.host, self.host_location, self.ping_response)
             sleep(self.host_timer - ((time() - self.starttime) % 1))
     

README.md

@@ -1,6 +1,6 @@
 # Docker Based Latency Monitor
 
-Docker container which tracks latency of one or many hosts and reports to InfluxDBv2.
+Docker container(s) which tracks latency of one or many hosts and reports to InfluxDBv2.
 
 ## Description
 
@@ -17,26 +17,45 @@ It´s based on python3 an makes usage of following python libraries:
 - configparser
 - time
 
+You can use it in *standalone* or *full stack* mode.
+
+**Standalone:**
+- Just the latency-monitor container which sends data to an external InfluxDB2 Server
+
+**Full Stack:**
+- Traefik container as Proxy (full TLS support)
+- InfluxDB2 container, fully setup and ready to take data
+- Grafana container, fully setup and connected (but without dashboards)
+- latency-monitor container sending data to the InfluxDB2 container
+
+
+## Requirements
+
+- Docker (CE)
+- Docker-Compose
+- InfluxDB Version >= 2
+- pythonping needs root privileges so same for the container
+
+## Configuration (GENERAL)
+
 Configuration can be passed via ENV **OR** configuration file.
 
 In case of using the ENV option you are just able to monitor **ONE** target for more targets please use the configuration file. 
 
 Also some influx connection options are just configurable via config file but normally they are not needed.
 
-Per default the python influx connector will cache all replies and sends them bundled every 30 seconds to the Influx DB.
 
-The container will be build at 1st start.
+### Behaviour 
+
+Per default the used python influxdb connector will cache all replies and sends them bundled every 30 seconds to the Influx DB.
+
+Actually the latency-monitor container is build on demand, a dockerhub image is on the roadmap...
 
 You can find everything under *./Docker_Build/* and in the python program itself [latency_monitor.py](./Docker_Build/latency_monitor.py)
 
-## Requirements
+-----
 
-- Docker
+### ENV Variables
-- Docker-Compose
-- InfluxDB Version >= 2
-- pythonping needs root privileges so same for the container
-
-## ENV Variables
 
 Name | Example | Usage | Option/Must
 :------: | :-----: | :-----: | :-----:
@@ -48,60 +67,111 @@ TARGET_HOST | 8.8.8.8 | Monitored Host (IP/FQDN) | must
 TARGET_TIMER | 3 | ping frequency in sec. | option
 TARGET_LOCATION | Google | decript. location | option
 
-## Config File
+-----
+
+### Config File
 
 **Instead** of using the ENV variables you can use a config file.
 
 **Keep in mind it´s a OR decision not a AND**  
 
-See [template_config.ini](./Docker_Build/template_config.ini)
+See [./latency-monitor/config.ini](./latency-monitor/config.ini)
 
-Rename the file to *config.ini* make your changes and add it as a volume mount to the container:
 
-### Docker-Compose Style
+#### Docker-Compose Style
+
+uncomment: 
 
 ```
-        volumes:
+# - ./latency-monitor/config.ini:/app/config.ini:ro # UNCOMMENT IF NEEDED
-            - /YOUR_PATH/config.ini:/app/config.ini
 ```
 
-### Docker-CLI Style
+#### Docker-CLI Style
 
 ```
-            docker latency-monitor -v /YOUR_PATH/config.ini:/app/config.ini
+docker latency-monitor -v ./latency-monitor/config.ini:/app/config.ini:ro
 ```
 
+-----
+-----
+
+## Configuration (Standalone)
+
+1st thing to do is creating the *docker-compose.yml* from [docker-compose-standalone.yml](./docker-compose-standalone.yml):
+
+```
+cp docker-compose-standalone.yml docker-compose.yml
+```
+
+### Variables
+
+Below paragraph:
+
+```
+####################################################
+# LATENCY-MONITOR
+####################################################
+```
+
+in the **.env** file *(env needs to be renamed to .env)* configure following variables:
+
+- YOUR_ORGANIZATION
+- YOUR_BUCKET_NAME
+- YOUR_ADMIN_TOKEN
+- YOUR_MONITORED_TARGET
+- YOUR_MONITORED_TARGET_TIMER
+- YOUR_MONITORED_TARGET_LOCATION
+
+### Lets go
+
+```
+docker-compose up -d latency-monitor
+```
+
+should do the job
+
 
 -----
 -----
 
 
-## Compose Files
+## Configuration (Full-Stack)
 
-### FULL-STACK
+### Easy peasy automatic mode
 
-1st thing to do is creating the *docker-compose.yml from [docker-compose-full_stack.yml](./docker-compose-full_stack.yml):
+Have a look at [./setup-full_stack.sh](./setup-full_stack.sh)
+
+Just create a valid *.env* File by:
 
 ```
-cp docker-compose-full_stack.yml docker-compose.yml
+cp env .env
 ```
 
-#### Certificate
+and edit it to your needs.
 
-*Traefik* will act as a proxy and ensures the usage of TLS so it needs your certificate and key file.
+After everyting within *.env* is in order just do:
-
-within the *docker-compose.yml* you will find:
 
 ```
-      - ./traefik/mycert.crt:/certs/cert.crt:ro
+./setup-full_stack.sh
-      - ./traefik/mycert.key:/certs/privkey.key:ro
 ```
 
-so please place your certificate file as *./traefik/mycert.crt* and the key file as *./traefik/mycert.key*.
+Everything should be right in place now.
 
-Thats it
+Just the certificates are missing look [here](#certificate)
+
+Now run it and mybe pick a example dashboard for grafana from [here](#grafana-dashboard-examples)
+
+-----
+-----
+
+### WTF manual mode
+
+REALLY???
+
+You need to set all on your own:
 
 #### Variables
+
 You need to configure Variables in following files to make the compose work:
 
 - **file**
@@ -112,30 +182,32 @@ You need to configure Variables in following files to make the compose work:
 -----
 
 - **docker-compose.yml** *(was docker-compose-full_stack.yml before)*
-    - PLACE_YOUR_FQDN_HERE (3 times)
+  - PLACE_YOUR_FQDN_HERE (3 times)
 
 -----
 
 - **.env** *(env needs to be renamed to .env)*
-    - YOUR_PATH_TO_CONTAINER_STATIC_DATA
+  - YOUR_PATH_TO_CONTAINER_STATIC_DATA
-    - YOUR_ADMIN_USER
+  - YOUR_ADMIN_USER
-    - YOUR_ADMIN_PASSWORD
+  - YOUR_ADMIN_PASSWORD
-    - YOUR_ORGANIZATION
+  - YOUR_ORGANIZATION
-    - YOUR_BUCKET_NAME
+  - YOUR_BUCKET_NAME
-    - YOUR_ADMIN_TOKEN
+  - YOUR_ADMIN_TOKEN
-    - YOUR_MONITORED_TARGET
+  - YOUR_MONITORED_TARGET
-    - YOUR_MONITORED_TARGET_TIMER
+  - YOUR_MONITORED_TARGET_TIMER
-    - YOUR_MONITORED_TARGET_LOCATION
+  - YOUR_MONITORED_TARGET_LOCATION
 
 -----
 
 - **grafana/provisioning/datasources/grafana-datasource.yml**
-    - YOUR_ADMIN_TOKEN
+  - YOUR_ADMIN_TOKEN
-    - YOUR_ORGANIZATION
+  - YOUR_ORGANIZATION
-    - YOUR_BUCKET_NAME
+  - YOUR_BUCKET_NAME
 
+-----
 
 #### File Permissions
+
 Because we are configuring *grafana* for permanent data storing and *grafana* actually runs with *UID* + *GID*: *472:472* it´s necessary to change permisson of die permanent storage directory we have configured.
 
 The directory build from the following config part of grafana within the docker-compose.yml:
@@ -156,60 +228,48 @@ then you have to do the following
 chown -R 472:472 /opt/docker/containers/grafana/var_lib
 ```
 
-#### Lets go
+Everything should be right in place now.
+
+Just the certificates are missing look [here](#certificate)
+
+Now run it and mybe pick a example dashboard for grafana from [here](#grafana-dashboard-examples)
+
+-----
+-----
+
+## Certificate
+
+*Traefik* will act as a proxy and ensures the usage of TLS so it needs your certificate and key file.
+
+within the *docker-compose.yml* you will find:
 
 ```
-docker-compose up -d
+      - ./traefik/mycert.crt:/certs/cert.crt:ro
+      - ./traefik/mycert.key:/certs/privkey.key:ro
 ```
 
-should do the job
+so please place your certificate file as *./traefik/mycert.crt* and the key file as *./traefik/mycert.key*.
+
+Thats it
 
 
-#### Grafana Dashboard Examples
 
-Within the local path *./examples/grafana/*  you can find example *.json* files which can be imported to grafana as dashboards to give you a first point to start with.
+
+
+
+## Grafana Dashboard Examples
+
+Within the local path [./examples/grafana/](./examples/grafana/) you can find example *.json* files which can be imported to grafana as dashboards to give you a first point to start with.
 
 
 -----
 -----
 
 
-### STANDALONE
-
-1st thing to do is creating the *docker-compose.yml from [docker-compose-standalone.yml](./docker-compose-standalone.yml):
-
-```
-cp docker-compose-standalone.yml docker-compose.yml
-```
 
 
-#### Variables
-
-Below paragraph: 
-
-```
-####################################################
-# LATENCY-MONITOR
-####################################################
-```
-
-in the **.env** file *(env needs to be renamed to .env)* configure following variables:
-
-- YOUR_ORGANIZATION
-- YOUR_BUCKET_NAME
-- YOUR_ADMIN_TOKEN
-- YOUR_MONITORED_TARGET
-- YOUR_MONITORED_TARGET_TIMER
-- YOUR_MONITORED_TARGET_LOCATION
 
 
-#### Lets go
-
-```
-docker-compose up -d latency-monitor
-```
-
-should do the job
 
 
 ## Authors
@@ -220,7 +280,16 @@ Contributors names and contact info
 
 ## Version History
 
-* 0.1
+* v0.2b
+  * cleanup
+
+* v0.2a
+    * fixed some missing variables
+    * fixe a missing integer declaration in latency-monitor
+    * added automatic config creation for full-stack
+    * cleanups
+
+* v0.1
     * Initial Release
 
 ## License

docker-compose-full_stack.yml

@@ -124,6 +124,10 @@ services:
     depends_on: 
       - influxdb
     restart: always
+    volumes:
+      # - ./latency-monitor/config.ini:/app/config.ini:ro # UNCOMMENT IF NEEDED
+      - /etc/timezone:/etc/timezone:ro
+      - /etc/localtime:/etc/localtime:ro
     environment: 
       - INFLUX_URL
       - INFLUX_TOKEN

docker-compose-standalone.yml

@@ -8,6 +8,10 @@ services:
     hostname: latency-monitor
     build: ./Docker_Build
     restart: always
+    volumes:
+      # - ./latency-monitor/config.ini:/app/config.ini:ro # UNCOMMENT IF NEEDED
+      - /etc/timezone:/etc/timezone:ro
+      - /etc/localtime:/etc/localtime:ro
     environment: 
       - INFLUX_URL
       - INFLUX_TOKEN

env

@@ -8,6 +8,7 @@
 
 MyPath=/YOUR_PATH_TO_CONTAINER_STATIC_DATA/
 
+MyFQDN=FQDN_OF_THIS_HOST
 
 ####################################################
 # INFLUXDB

latency-monitor/config.ini

@@ -0,0 +1,35 @@
+# Config File Usage
+# 
+# all MUST options are marked with #!!! and have to be configured
+#
+# Config file ist IGNORED if equivalent environment variables are passed 
+#
+#
+
+
+[influx] # SOME are MUST; influxdb2 connection relevant options
+#!!! infx_url = http://YOUR_INFLUX_DB_V2:8086
+#!!! infx_token = YOUR_INFLUX_DB_V2_API_TOKEN
+#!!! infx_bucket = YOUR_INFLUX_DB_V2_BUCKET
+#!!! infx_org = YOUR_INFLUX_DB_V2_ORGANIZATION
+# infx_batch_size = 60 # OPTIONAL; default is '60'
+# infx_flush_interval = 30_000 # OPTIONAL; default is '30_000'
+# infx_jitter_interval = 5_000 # OPTIONAL; default is '5_000'
+# infx_retry_interval = 5_000 # OPTIONAL; default is '5_000'
+
+
+[hosts] # ONE HOST is MUST, rest ist optional; IP or FQDN
+#!!! host1 = 8.8.8.8
+# host2 = 8.8.4.4
+# host3 = 1.1.1.1
+
+[hosts_timer] # OPTIONAL; ping interval in seconds; default is '5'
+# host1 = 1
+# host2 = 3
+# host3 = 5
+
+[hosts_location] # OPTIONAL; location of host; default is 'unknown'
+# host1 = Google-DNS1
+# host2 = Google-DNS2
+# host3 = Cloudflare-DNS
+

setup-full_stack.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+
+Date=`date +%Y%m%d_%H%M%s`
+
+# locate my path
+MyScriptPath=`dirname $0`
+MyScriptPathContainer="$MyScriptPath/CONTAINER/"
+
+# Check if .env allready exists
+if ! test -e $MyScriptPath/.env; then
+    echo "FAIL: You need to copy file env to .env and edit it!!!"
+    exit 1
+fi
+
+# backup old compose files
+cp -f $MyScriptPath/docker-compose.yml $MyScriptPath/docker-compose-$Date.backup
+# copy compose template to final compose file (OVERWRITTEN!!!) 
+cp -f $MyScriptPath/docker-compose-full_stack.yml $MyScriptPath/docker-compose.yml
+
+# Make relevant grafana templating direcotries
+echo "MKDIR: creating $MyScriptPath/grafana/provisioning/datasources"
+mkdir -p $MyScriptPath/grafana/provisioning/datasources
+# backup old grafana datasource file
+cp -f $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml $MyScriptPath/grafana/grafana-datasource-$Date.backup
+# copy grafana datasource file template to grafana datasource file
+cp -f $MyScriptPath/grafana/grafana-datasource-template.yml $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml
+
+
+# Replace .env MyPath Path with local path if NOT changed
+sed -i -e "s#/YOUR_PATH_TO_CONTAINER_STATIC_DATA#$MyScriptPathContainer#g" $MyScriptPath/.env
+
+
+# Read variables from .env file
+source $MyScriptPath/.env
+
+echo "INFO: MyPath is $MyPath"
+
+# Make relevant direcotries
+echo "MKDIR: creating $MyPath"
+mkdir -p $MyPath
+echo "MKDIR: creating $MyPath/influxdb/"
+mkdir -p $MyPath/influxdb/
+echo "MKDIR: creating $MyPath/grafana/var_lib"
+mkdir -p $MyPath/grafana/var_lib
+
+# Changes in docker-compose.yml
+echo "CHANGE: replace PLACE_YOUR_FQDN_HERE with $MyFQDN in $MyScriptPath/docker-compose.yml"
+sed -i -e "s/PLACE_YOUR_FQDN_HERE/$MyFQDN/g" $MyScriptPath/docker-compose.yml
+
+# Changes in grafana/provisioning/datasources/grafana-datasource.yml
+echo "CHANGE: replace YOUR_INFLUXDB_URL with $INFLUX_URL in $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml"
+sed -i -e "s#YOUR_INFLUXDB_URL#$INFLUX_URL#g" $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml
+echo "CHANGE: replace YOUR_ADMIN_TOKEN with $INFLUX_TOKEN in $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml"
+sed -i -e "s/YOUR_ADMIN_TOKEN/$INFLUX_TOKEN/g" $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml
+echo "CHANGE: replace YOUR_ORGANIZATION with $INFLUX_ORG in $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml"
+sed -i -e "s/YOUR_ORGANIZATION/$INFLUX_ORG/g" $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml
+echo "CHANGE: replace YOUR_BUCKET_NAME with $INFLUX_BUCKET in $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml"
+sed -i -e "s/YOUR_BUCKET_NAME/$INFLUX_BUCKET/g" $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml
+
+# Correct owner and permissions to satisfy the containers
+echo "CHMOD: chmod -R 755 $MyPath"
+chmod -R 755 $MyPath
+echo "CHMOD: chmod -R 755 $MyScriptPath/grafana/provisioning"
+chmod -R 755 $MyScriptPath/grafana/provisioning 
+echo "CHMOD: chmod 644 $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml"
+chmod 644 $MyScriptPath/grafana/provisioning/datasources/grafana-datasource.yml
+echo "CHOWN: chown -R 472.472 $MyPath/grafana/var_lib"
+chown -R 472.472 $MyPath/grafana/var_lib
+
+echo "OK: All done"
+