Central Logging Revisited

In my former post about collecting logs from running containers, I thought fluentd was a good choice as it was so much lighter then logstash as a log shipper and 'enricher'. But as I was trying to parse postfix logs from a mailserver container for which exists some very nice grok patterns, available on github and I did not have the time (and maybe motivation) to do all that work somehow again for fluentd, I switched back to logstash. And use the syslog log driver  to ship the logging from the containers to logstash (which is running in a container)

I use the sebp/elk container as a base. It also contains the elasticsearch and kibana services and configure the patterns/and configs from a directory mounted into the container.

# elastic stack

  elk:
    build: elk
    networks:
      - default
      - traefik_proxy
    restart: always
    ports:
      - "127.0.0.1:5000:5000"
    environment:
      TZ: ${TZ}
    labels:
      - "traefik.enable=true"
      - "traefik.backend=elk"
      - "traefik.port=5601"
      - "traefik.frontend.rule=Host:log.${DOMAINNAME}"
      - "traefik.frontend.auth.basic.usersFile=/shared/.htpasswd"
      - "traefik.docker.network=traefik_proxy"
    volumes:
      - elk-data:/var/lib/elasticsearch
      - ${DOCKERDIR}/elk/conf.d:/etc/logstash/conf.d
      - ${DOCKERDIR}/elk/patterns.d:/etc/logstash/patterns.d

I also restrict the logstash input port to localhost so it is not exposed for the whole world. The traefik reverse proxy exposes the kibana logviewer.

If I want the log from a container to go to the elk container. I only have to include the following in the compose part of the service:

    logging:
      driver: "syslog"
      options:
        syslog-address: "tcp://localhost:5000"
        tag: "{{.Name}}"