Logging Made Easy: Docker, Syslog, and Privacy-Focused Analytics

In my previous home lab post I noted that I use Docker to run most of my services. That now includes this site in a container I am building, pushing, and deploying automatically whenever I push a commit. (I will need to write a post about that too)

I also set up Goatcounter * to do some anonymous tracking of visitors to this site. I mostly want to know about referrers and what content people were finding interesting. I decided I wanted to forgo the JS tracking script in favor of just using server counts.

Goatcounter supports this out of the box. It expects to either import data occasionally or it can be set to tail a log file. Since by default Docker does not create standard log formats, I had to learn a bit about how to get Docker to send logs to a text file.

I ended up using rsyslog and some tweaks to my container configuration. The result is a log file for my container, the ability to easily add this same sort of logging to any other container on the host machine, and a Goatcounter service that is sending anonymized data to goatcounter.

First, you need to setup the logging mechanism in rsyslog.

Create the file /etc/rsyslog.d/30-docker.conf

$FileCreateMode 0644
$template DockerDaemonLogFileName,"/var/log/docker/docker.log"
$template DockerContainerLogFileName,"/var/log/docker/%SYSLOGTAG:R,ERE,1,FIELD:docker/(.*)\[--end:secpath-replace%.log"
if $programname == 'dockerd' then {
?DockerDaemonLogFileName
stop
}
if $programname == 'containerd' then {
?DockerDaemonLogFileName
stop
}
if $programname == 'docker' then {
if $syslogtag contains 'docker/' then {
?DockerContainerLogFileName
stop
}
}
$FileCreateMode 0600

You will want to reload rsyslog after creating that file sudo systemctl restart rsyslog

The file creates a new docker.log file for system log messages. It will also create a file [field].log where [field] is replaced by the second part of the log_opts tag.

In terraform you get this by adding log_opts to your container, along with setting the log_driver as syslog

resource "docker_container" "containername" {
  name  = "containername"
  image = "registry.me/imagename:latest"

...

  log_opts = {
    tag = "docker/containername"
  }

  log_driver = "syslog"

...
}

Or on the command line

docker run -d \
  --name containername \
  --log-driver=syslog \
  --log-opt tag=docker/containername \
  registry.me/imagename:latest

Now when this container starts, any log messages it sends to stdout or stderr are logged to /var/log/docker/containername.log

I also recommend setting up some log rotation for those files so you do not fill your filesystem if you have noisy containers.

Now create /etc/logrotate.d/rsyslog-docker to have logrotate monitor and rotate my logs.

/var/log/docker/*.log
{
weekly
rotate 10
minsize 200M
missingok
notifempty
compress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}

Finally, I downloaded the goatcounter binary and told it to tail the log. /etc/systemd/system/goatcounter-blog.service.

[Unit]
Description=GoatCounter Import Service
After=network.target

[Service]
Environment=GOATCOUNTER_API_KEY=[your key]
User=[user]
WorkingDirectory=/home/[user]
ExecStart=/home/[user]/goatcounter import -follow -format=combined -exclude=static -exclude='path:glob:/fonts/**' -site='https://goatcounter.com' /var/log/docker/containername.log
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

You will need to enable this service with

sudo systemctl enable goatcounter-blog.service
sudo systemctl start goatcounter-blog.service

Now, assuming the log files created by the container match the Combined Log Format, you will see visits added to your Goatcounter statistics.

One caveat to this approach is that Docker does not give you an easy way to differentiate between stderr and stdout so all messages go to your single log file. Ideally there would be a way to differentiate between the two messages types and send them to a .log or a .error file, but that does not appear to be possible today.

*If you want to self host Goatcounter yourself, take a look at my Docker container