ELK配置笔记

filebeat安装配置

1 下载,安装

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.4.3-linux-x86_64.tar.gz

mv filebeat-8.4.3-linux-x86_64.tar.gz   /opt
cd /opt
tar -zxvf filebeat-8.4.3-linux-x86_64.tar.gz
cd filebeat-8.4.3-linux-x86_64

2 配置

修改filebeat.yml

# ========================= Filebeat inputs ======================
filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true      #设为true,启用这个input配置

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log      #日志路径
    #- c:programdataelasticsearchlogs*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1
 processors:
    #- add_host_metadata:
    #    when.not.contains.tags: forwarded
    #- add_cloud_metadata: ~
    #- add_docker_metadata: ~
    #- add_kubernetes_metadata: ~
    - drop_fields:   #剔除不需要的字段，减少数据量
        fields: ["log","input","ecs","agent","tags"]
        ignore_missing: false

  # =================== Outputs ===============================
  # ---------------------------- Elasticsearch Output ----------------------------
  #本配置传给logstash,所以本部分全部注释掉
  ...
  # ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["192.168.0.100:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

3启动

#启动
nohup ./filebeat -e -c filebeat.yml >/dev/null 2>&1 &

#指定配置文件启动
./filebeat run -e -c filebeat.yml -d "publish"

4 清除标记

Filebeat会将文件读取位置记录 /opt/filebeat-8.4.3-linux-x86/data/registry 文件夹中，想重新从文件开始读取需要删除 registry 文件夹，然后重启Filebeat。