How to Manage Docker Container Logs?

Managing Docker container logs is very important for keeping our applications healthy and running well. Docker container logs give us, the developers and system admins, important information about how our applications behave. They also show us errors and help us see how the system performs. If we manage these logs properly, they will not take too much disk space. We can also find important information easily when we need to fix problems or analyze data.

In this article, we will talk about good ways to manage Docker container logs. We will look at Docker logging drivers, how to set up logging options, how to view logs, how to rotate logs automatically, and how to use the ELK Stack for centralizing logs. By the end of this guide, we will understand how to manage Docker container logs in a smart way. Here are the key points we will cover:

  • How to Effectively Manage Docker Container Logs
  • What Are Docker Logging Drivers
  • How to Configure Docker Logging Options
  • How to View Docker Container Logs
  • How to Rotate Docker Logs Automatically
  • How to Centralize Docker Logs with ELK Stack
  • Frequently Asked Questions

If you want to learn more about Docker and what it can do, we can check these articles: What is Docker and Why Should You Use It?, What Are Docker Images and How Do They Work?, and What is Containerization and How Does It Relate to Docker?.

What Are Docker Logging Drivers?

Docker logging drivers help us manage the logs from containers. They decide how we collect, store, and send logs. By default, Docker uses the json-file logging driver. This driver writes logs in JSON format to a file on our host system.

Common Docker Logging Drivers

  1. json-file: This is the default driver. It logs container output in JSON format.
  2. syslog: This driver sends logs to the syslog daemon on the host.
  3. journald: This one works with systemd’s journal for storing logs.
  4. gelf: It sends logs to a Graylog Extended Log Format (GELF) endpoint.
  5. fluentd: This driver routes logs to Fluentd for flexible log collection.
  6. awslogs: It sends logs to Amazon CloudWatch.
  7. splunk: This driver forwards logs to a Splunk server.

Configuring Logging Drivers

To set a logging driver for a container, we can use the --log-driver option with the docker run command. Here is an example using the syslog driver:

docker run --log-driver=syslog --log-opt syslog-address=udp://localhost:514 my-container

Viewing Logs Based on Driver

  • For the json-file, we can view logs by using:

    docker logs <container_id>

  • For syslog, we can access logs through the syslog daemon.

Setting Default Logging Driver

To set a default logging driver for all containers, we need to change the Docker daemon configuration file (/etc/docker/daemon.json):

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

After we edit this file, we should restart the Docker service:

sudo systemctl restart docker

By knowing and setting up Docker logging drivers well, we can handle container logs in a way that fits our application needs and setup. To learn more about Docker and how it works, we can visit What is Docker and Why Should You Use It?.

How to Configure Docker Logging Options?

We can set up Docker logging options to control how logs are collected and stored from our Docker containers. We can choose logging drivers and set log options in our docker run command or in the Docker daemon configuration.

Using docker run

When we start a container, we can configure logging options with the --log-driver and optional --log-opt flags:

docker run \
  --log-driver=<driver_name> \
  --log-opt <option1>=<value1> \
  --log-opt <option2>=<value2> \
  <image_name>

Example: Using the JSON File Driver

The default logging driver is json-file. If we want to change the logging options, we can use:

docker run \
  --log-driver=json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  nginx

This setup limits each log file to 10 MB. It keeps a maximum of 3 log files.

Configuring Logging in Docker Daemon

We can also set logging options for all containers in the Docker daemon configuration file. This file is usually found at /etc/docker/daemon.json.

Example Configuration:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

After we change the configuration file, we need to restart the Docker service to apply the changes:

sudo systemctl restart docker

Available Logging Drivers

  • json-file: This is the default driver that saves logs in JSON format.
  • syslog: This sends logs to the syslog daemon.
  • journald: This uses the journald logging service.
  • gelf: This sends logs to a Graylog Extended Log Format (GELF) endpoint.
  • fluentd: This sends logs to Fluentd.
  • awslogs, splunk, etc.: These are for third-party logging services.

Checking Current Logging Configuration

If we want to see the current logging options for a running container, we can use:

docker inspect <container_id> --format='{{.HostConfig.LogConfig}}'

This command shows details about the logging driver and options that are being used for the specified container.

By setting Docker logging options right, we can manage and keep our container logs well. This way, we make sure logging fits our application’s needs.

How to View Docker Container Logs?

We can view Docker container logs using the Docker CLI with the docker logs command. This command helps us get logs from a specific container. It lets us check output and fix issues.

Basic Command Syntax

docker logs [OPTIONS] CONTAINER

Common Options

  • -f, --follow: This option lets us see new logs as they come in.
  • --tail: This option shows only the last N lines of logs.
  • --since: This option shows logs from a specific time.

Examples

  1. View logs of a specific container:

    docker logs my_container

  2. Follow logs in real-time:

    docker logs -f my_container

  3. Show the last 100 lines of logs:

    docker logs --tail 100 my_container

  4. Show logs since a specific time:

    docker logs --since 1h my_container

Viewing Logs from All Containers

To see logs from all running containers, we can use docker ps and docker logs commands in a loop:

for container in $(docker ps -q); do
    echo "Logs for container: $container"
    docker logs $container
done

Accessing Logs for Stopped Containers

To get logs for a stopped container, we just use the same docker logs command with the container ID or name:

docker logs my_stopped_container

Using these ways will help us manage and view Docker container logs better. This is important for checking and fixing our applications. For more information about Docker management, we can check what is Docker and why should you use it.

How to Rotate Docker Logs Automatically?

We need to manage Docker container logs well. Rotating logs automatically helps us save disk space and keeps the logging system fast. Docker has built-in options for log rotation. We can use the json-file logging driver. This driver is the default one.

Configuring Log Rotation

We can set up log rotation by adding options in the docker run command. We can also do it in the Docker daemon’s configuration file. Here are some common options for log rotation:

  • max-size: This is the biggest size for the log file before it gets rotated.
  • max-file: This is the most number of log files we want to keep.

Example: Using docker run

We can add log rotation options when we run a container like this:

docker run --log-driver=json-file \
           --log-opt max-size=10m \
           --log-opt max-file=3 \
           your-image

In this example, each log file can be up to 10 MB. Docker will keep a maximum of 3 log files.

Configuring via Docker Daemon

If we want to set log rotation for all containers, we need to change the Docker daemon configuration file. This file is usually at /etc/docker/daemon.json. We add or change this configuration:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

After we change the configuration file, we must restart the Docker service. This is how we do it:

sudo systemctl restart docker

Verifying Log Rotation

To see if log rotation is working, we check the logs of a running container. We can look at the log files in the /var/lib/docker/containers/<container-id>/ folder.

This setup helps us manage Docker container logs better. It automatically rotates them and keeps our system clean and fast. For more information about Docker, we can check out what Docker is and why you should use it.

How to Centralize Docker Logs with ELK Stack?

We can centralize Docker container logs using the ELK Stack. ELK stands for Elasticsearch, Logstash, and Kibana. This setup helps us manage and analyze logs from many containers easily. Let us see how to set it up.

  1. Install ELK Stack: We can run the ELK stack using Docker. Use this docker-compose.yml to set up Elasticsearch, Logstash, and Kibana.
version: '3'
services:
  elasticsearch:
    image: elasticsearch:7.10.1
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"

  logstash:
    image: logstash:7.10.1
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5044:5044"

  kibana:
    image: kibana:7.10.1
    ports:
      - "5601:5601"
  1. Configure Logstash: Next, we need to create a logstash.conf file. This file will tell Logstash where to get the input and where to send the output. It will read logs from Docker containers and send them to Elasticsearch.
input {
  tcp {
    port => 5044
    codec => json_lines
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "docker-logs-%{+YYYY.MM.dd}"
  }
}
  1. Run ELK Stack: Now, we run the stack. Go to the folder where your docker-compose.yml is. Then run:
docker-compose up -d
  1. Configure Docker Logging Driver: We need to send logs from Docker containers to Logstash. To do this, we set the logging driver to gelf in the container settings.
docker run -d \
  --log-driver=gelf \
  --log-opt gelf-address=udp://<logstash-ip>:5044 \
  your_docker_image

Make sure to replace <logstash-ip> with the IP address of your Logstash container or use localhost if you run it on your machine.

  1. Access Kibana: Open your web browser. Go to http://localhost:5601. Here, we can see and analyze logs in Kibana.

  2. Create an Index Pattern: In Kibana, go to the “Management” section. Then click on “Index Patterns.” Create an index pattern for docker-logs-*. This will help us start analyzing our logs.

This setup helps us centralize Docker container logs. Now we can monitor and troubleshoot our applications better. For more advanced logging methods, we can look into how to manage Docker container logs with other tools and settings.

Frequently Asked Questions

1. What are Docker container logs used for?

We use Docker container logs to watch and fix problems in applications that run in containers. These logs show the output from the processes inside the container. This includes standard output (stdout) and standard error (stderr). When we manage Docker container logs well, we can see how the application works, find issues, and make it run better. For more about Docker’s setup, check out this article on Docker’s core parts.

2. How can I view Docker container logs?

We can view Docker container logs using the command line. Just run the command docker logs <container_id> to see the logs for a specific container. This command shows the logs in real-time. It is very important for fixing issues. We can also use flags like -f to follow logs or --tail to limit how many lines we see. For more on Docker container actions, look at this guide on how to create and manage Docker containers.

3. What are Docker logging drivers, and how do they work?

Docker logging drivers tell us how to collect and save logs from Docker containers. There are different logging drivers like json-file, syslog, and fluentd. They give us different ways to manage logs. Picking the right logging driver is important for good log management and to work well with other logging systems. For more details about containerization, see this article on what is containerization and how it relates to Docker.

4. How do I configure Docker logging options?

To set Docker logging options, we need to choose the logging driver and settings in the Docker daemon config file (/etc/docker/daemon.json). For example, we can set the default logging driver to json-file and add options for log rotation. After we change the settings, we need to restart the Docker service for the changes to work. For a full guide on installing Docker, visit this article on how to install Docker on different operating systems.

5. How can I centralize Docker logs using the ELK stack?

We can centralize Docker logs using the ELK stack (Elasticsearch, Logstash, and Kibana). This helps us search, analyze, and visualize logs better. We can set Docker to send logs to Logstash with a compatible logging driver. Once the logs are in Elasticsearch, we can use Kibana to visualize and query our logs easily. This makes our log management much better. For more about what Docker can do, check out this article on the benefits of using Docker in development.