devops,

Docker - Logging and Metrics

Sumeet Sumeet Follow Jul 02, 2020 · 9 mins read
Docker - Logging and Metrics
Share this

Logging vs Metrics

Those who are into operations or even those who are involved in any kind of debugging an application, they know the importance of monitoring and event management. Monitoring is a fundamental area to any production environment as even a downtime of a few seconds can have a direct and huge impact on business. Thus, it is of utmost importance to have monitoring of production application in place. More so, it is important to have a great operations team to understand the monitoring logs and know where exactly to apply the fix in order to minimise the impact. Logs can be of different types - application, system, server, database etc. and same goes with the metrics.

Often, operations rely heavily on logs and metrics to get to the root cause of the issue. Most of us have already dealt with debugging at some point of time in our lives. There is an entire section of the IT industry that dedicates itself into logs and metrics management at various levels. But how do we get to the logs and metrics produced by containers? Are they even accessible to us? Can we customize the information being logged? and many more similar questions can be asked. Before we get to those let’s answer this - But are they really different?

Logs vs Metrics

Logs and metrics are not the same thing. Logs usually provide you with a verbose information about the running application or environment. They can be event driven or scheduled. Logs produced by various applications can have different structures, thus gathering logs from various sources results in a huge chunk of unstructured data. A simple example of log could be request-response logs - which is usually logged by a web or application server in order to track incoming traffic from various external sources as well as outgoing response. Requests may originate from anywhere on the internet. Volume of logs produced directly depends on the number of incoming requests. A developer of the application or microservice may choose to log certain key details about the request like source IP address, timestamp, certificate information etc. However, a developer of another application may choose some other aspects of this request as key details to be logged giving rise to the unstructured nature of system logs. Also, due to the amount of data being produced just by logging, storage soon becomes an issue. However, due to their verbose nature it becomes easier to trace the problem in the system. In fact once these logs are structured in a single place and indexed, it enables us to have any kind of view of the system.

Metrics on the other hand are based on timed data points. These are essentially numbers captured at a regular interval of time. Every metric which is captured also has a label associated with it which indicates the aspect being measured. Due to this structured, minimal and labeled approach, it becomes very easy to create visualisations which instantly make sense. Imagine, you are hosting a web application which receives a huge amount of traffic on weekends. Due to their nature, the amount of storage consumed while recording these metrics will always be the same - an incremented number. Thus, you don’t always run into storage issues. However, to know the system better from various viewpoints, you may want to add more metrics, which would mean addition of more time series numbers. This is a much better solution in narrowing down the problem area especially in the cases where the storage pricing model is based on /GB/month. However, as a downside it will not be very helpful in identifying the exact root cause itself.

Logs and metrics are not the same but they do not address monitoring and event management challenges in their entirety. ELK and Containers are examples of log processing tools, where they make use of logstashes and indexing. I did a post sometime back on Splunk, you can check the same here. Prometheus is an example of a metrics generation tool, which we shall go into more details.

Logging in Docker containers

As we now know, Docker containers run individual environments where applications are hosted. All the logs are generated within this container and yes, there is a straightforward way to see these logs by running below commands. docker logs outputs the command for a container and ‘docker service logs` outputs the commands of services. Services are Docker containers running in swarm mode where it is easier to manage multiple containers.

docker logs <container_name>
docker service logs <service_name>

Linux OS normally use 3 types of file descriptors for I/O, namely - STDIN (for input via keyboard or other commands), STDOUT (for standard output to terminal) and STDERR (to output error messages). By default, docker logs uses outputs from STDOUT and STDERR to log it’s output. However, you can also customize the log output by using logging drivers. The default logging driver is json-file. A simple docker info command can verify the same. (Screenshot below)

Logs vs Metrics

Docker uses the same to store logs unless specified differently. Some examples of logging drivers are as below. Complete list of logging drivers for Docker containers can be found here.

  • local - for local logging
  • awslogs - to push logs to Amazon CloudWatch Logs
  • splunk - to push logs to Splunk
  • syslog - to write logs to syslog

Which log driver to use, can be specified in a file /etc/docker/daemon.json. You can add all those options which you normally use to run a Docker container as key-value pairs in this file instead of specifying all the options in the run command every time. Along with this information, you can specify additional options with respect to logging using log-opts key value pairs. Every logging driver has their own set of optional options to be set.

Be aware that Docker community edition supports logging either using docker logs only or logging drivers only. Enterprise edition supports dual logging.

In addition to logging drivers, you may also want to use 3rd party logging driver plugin. The external logging implementation needs to be converted into Docker image and hosted on Docker hub from where the image can be pulled and installed on the host where the docker is hosted. Using these custom logging plugins within Docker is similar to the way logging drivers are used in daemon.json file. There are many more options which can be configured when it comes to Docker container logging, but these are the basics to start with.

Docker metrics

Docker integrates seamlessly with Prometheus for gathering various container metrics. Prometheus is an open source monitoring and alerting solution which implements dimensional data, performs powerful queries, offers great visualization capabilities, has efficient storage mechanisms and offers many integrations.  I am sure Prometheus is awesome, but that is a topic for sometime later.

Getting up and running with Prometheus for Docker is very easy. All you need to do is spin a Prometheus docker service and feed the Prometheus configuration file to this service. This document explains the steps to do the same. The thing to note here is, Prometheus service listens on port 9323, so you would have to direct Docker daemon by specifying below settings in the daemon.json file.

{
    "metrics-addr: "127.0.0.1:9323",
    "experimental": true
}

Restart the docker daemon for these changes to take effects. Below image represents what we have achieved here. We have live application containers along with Prometheus service running on the host. Docker daemon is configured to push all the metrics to Prometheus service on port 9323.

Logs vs Metrics

Headover to the Prometheus dashboard by visiting http://0.0.0.0:9090/targets from your browser and you would be able to see something like below in the Graph tab. I spun several docker service ping jobs for a brief period of time and there you can see the prometheus_engine_query_duration_seconds metric giving me the metric data. Similarly, there are tons of other useful metrics available out of the box which help you monitor critical metrics for all the containers and services running on the host. You can even create your own custom metrics like this using Prometheus as well.

Logs vs Metrics

Well, that’s a wrap for this post. My intention here was to provide brief pointers towards logging and monitoring in Docker containers. Hope that was helpful. If yes, do subscribe!

Join Newsletter
Get the latest news right in your inbox!
Sumeet
Written by Sumeet Follow
Hi, I am Sumeet, and I believe the world belongs to the doers. Here, I publish my technical tinkering experiences. I hope you like it!