Docker Application Logging Using Prometheus

Docker logs by themselves reveal little about your containerized application. To get a true monitoring strategy in place, that includes your application metrics, you need a tool like Prometheus. This article shows a quick way to

• a) Setup prometheus on the host or inside a container.
• b) Configure your App to post to a prometheus endpoint and
• c) Non Prometheus based commands for inspecting running containers.

Prometheus – Standalone or as a container ?

Prometheus can be run as a  standalone exe on the host OR can run it inside a docker container. The recommendation is to run it inside a container; however, if you are not running a swarm of services, you could just  set it up on the host.

Prometheus on the Host – Setup

1. Download and extract prometheus.exe; add c:\prometheus to your path

https://github.com/prometheus/prometheus/releases/download/v2.7.1/prometheus-2.7.1.windows-amd64.tar.gz

2. Create default YAML file

3. Start it using –

prometheus --config.file "\tmp\prometheus.yml"

4.  Test localhost:9090/graphs

Prometheus as a container – Setup

docker run -p 9090:9090 prom/prometheus

5. https://www.prometheusbook.com/MonitoringWithPrometheus_sample.pdf

Downside of Prometheus based Logging

Prometheus , due to it’s constant calculation engine,  uses between 8 and 10 GB RAM (whether running on the host or in a container).  This may not be a feasible option for several server VMs, which have little extra memory to spare.

To Log App Specific Metrics – Application Specific Setup

To expose the /metrics endpoint from your application, you  need to install prometheus client libraries for your app (as shown in the URL below).

https://coreos.com/operators/prometheus/docs/latest/exposing-metrics.html

Summary

Appendix : Quick Inspection of Containers (without Prometheus) – Inspecting Running Containers and Viewing Event Logs

To connect to docker and launch a PS prompt inside the container

docker exec CONTAINERID

To Inspect a folder or a file inside the container

docker exec containeriD | ls <dir path>

To List all the processes inside the container

    First connect to it using docker exec and launch a PS prompt

 docker exec <container> | PS 

    Then, just use Powershell’s

        List-processes 

To view the event log inside the container (once you have a PS prompt using     docker exec <container> | PS  )

get-eventlog system -newest 10000 | format-table -wrap > c:\data\syslog.txt (if c:\data is defined as a volume, it can be read from the host) 

# this shows source as ‘Docker’ but can change you ‘Application’ or custom

Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-5) | Sort-Object Time 

#To search for specific terms in an event log

Get-WinEvent –listlog *mykeyword*

To Copy files from host to container

 docker cp foo.txt mycontainer:/foo.txthttp:

To Inspect all the properties (volumes, ports…) of a running container

First get the ip address of the running container – lookup the container id (docker container ls)

docker inspect c5e8de7faf05 - will give you the IP

docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $INSTANCE_ID