Notes from my experiment with using ELK stack + Packetbeat for network monitoring.

Kick off the stack

I have tweaked the docker-compose.yml file from Elastic search documentation [1].

I have added health checks to Elastic & Kibana containers and added Packetbeat that depends on these two to be running and “be healthy”.

docker-compose.yml:

version: '2.2'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
    healthcheck:
      test: ["CMD", "curl","-s" ,"-f", "http://localhost:9200/_cat/health"]
      interval: 30s
      #start_period: 60s


  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    ports:
      - 9201:9201
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    ports:
      - 9202:9202
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.6.2
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic
    healthcheck:
      test: ["CMD", "curl", "-s", "-f", "http://localhost:5601/"]
      retries: 10

  pktb01:
    image: docker.elastic.co/beats/packetbeat:7.6.2
    container_name: pktb01
    user: packetbeat
    #volumes:
    #  - ./packetbeat.docker.yml:/usr/share/packetbeat/packetbeat.yml:ro,z
    cap_add:
      - NET_RAW
      - NET_ADMIN
    network_mode: host
    environment:
      ELASTICSEARCH_HOSTS: localhost:9200
    command: >
              packetbeat
                 -e
                 -strict.perms=false
                 -E setup.kibana.host="localhost:5601"
                 -E setup.dashboards.enabled=true
    restart: on-failure
    depends_on:
      es01:
        condition: service_healthy
      kib01:
        condition: service_healthy

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

The setup as prepared doesn’t require any additional files/directories to be mounted to the containers. Necessary config options are provided via environment variables or directly via “command” as command line arguments (for Packetbeat).

If you want to provide a customized Packetbeat config file, just prepare it (./packetbeat.docker.yml) and un-comment the “volumes” section for Packetbeat service (pktb01).

Then kick off the stack by:

$ sudo docker-compose up

Configure Kibana

Our Kibana will have pre-loaded example Packetbeat dashboards because of the “setup.dashboards.enabled” config option. We just need to create an index with pattern “packetbeat-*” and add then we can use the pre-defined dashboards, for example “[Packetbeat] Overview ECS” that provides the hi-level overview.

Debugging tips

Check what network interfaces the packetbeat sees:

$ sudo docker exec pktb01 packetbeat devices
0: wlp61s0 (No description available) (192.168.1.102 fe80::c5b5:6c56:c76d:f8ec)
1: veth479ec45 (No description available) (fe80::40b1:1cff:fe75:66a3)
2: br-dd3d8fead435 (No description available) (172.18.0.1 fe80::42:84ff:fe04:9b57)
3: veth08b685b (No description available) (fe80::94c6:c2ff:fe02:dcf7)
4: vethf13336f (No description available) (fe80::4884:b9ff:fed9:8552)
5: veth9686383 (No description available) (fe80::1850:b2ff:fe06:17e1)
6: any (Pseudo-device that captures on all interfaces) (Not assigned ip address)
7: lo (No description available) (127.0.0.1 ::1)
8: virbr0 (No description available) (192.168.122.1)
9: docker0 (No description available) (172.17.0.1)
10: enp0s31f6 (No description available) (Not assigned ip address)
11: nflog (Linux netfilter log (NFLOG) interface) (Not assigned ip address)
12: nfqueue (Linux netfilter queue (NFQUEUE) interface) (Not assigned ip address)

References

[1] https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html