资讯

展开

【老男孩Linux技术分享】5分钟带你搞懂日志采集利器Filebeat!

作者:快盘下载 人气:

  filebeat是用于"转发"和"集中日志数据"的轻量级数据采集器

  filebeat会监视指定的日志文件路径,收集日志事件并将数据转发到elasticsearch,logstash,redis,kafka存储服务器。

  当您要面对成百上千,甚至成千上万的服务器,虚拟机的容器生成的日志时,请告别SSH吧。

  Filebeat将为您提供一种轻量级方法,用于转发和汇总日志与文件,让简单的事情不再繁杂。

【老男孩Linux技术分享】5分钟带你搞懂日志采集利器Filebeat!

  | Filebeat的组件

  Filebeat包含两个主要组件,input(输入)和Harvester(收割机),两个组件协同工作将文件的尾部最新数据发送出去。

  Harveste组件:负责逐行读取单个文件的内容,然后将内容发送到输出。

  input组件:输入负责管理收割机并找到所有要读取的源。该参数的源文件路径需要使用者手动配置。

  Spooler(缓冲区):将Harvester组件采集的数据进行统一的缓存,并发往目的端,可以是 Elasticsearch, Logstash , kafka 和 Redis 等。

  | Filebeat工作原理

  filebeat工作流程如下:

  1、filebeat启动后,filebeat通过Input读取指定的日志路径;

  2、为该文件日志启动收割进程harvester,每个收割进程读取一个日志文件的新内容,并发送这些新的日志数据到处理程序spooler;

  3、spooler会集合这些事件,最后filebeat会发送集合的数据到你指定的位置。

  Filebeat如何保持文件的状态?

  Filebeat保持每个文件的状态,并经常将状态刷新到注册表文件(data/registry/filebeat/log.json)中的磁盘。

  该状态用于记住收割机读取的最后一个偏移量,并确保发送所有日志行。

  Filebeat如何确保至少一次交付?

  Filebeat保证事件将至少传送到配置的输出一次并且不会丢失数据。

  Filebeat能够实现这种行为,因为它将每个事件的传递状态存储在注册表文件中。

  部署Filebeat环境

  | 安装Filebeat软件

# 编译安装Filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.1-linux-x86_64.tar.gz

tar xf filebeat-7.12.1-linux-x86_64.tar.gz -C /oldboyedu/softwares/

cd /oldboyedu/softwares/

ln -s filebeat-7.12.1-linux-x86_64 filebeat

vim /etc/profile.d/filebeat.sh

# 添加Filebeat的环境变量

cat /etc/profile.d/filebeat.sh

#!/bin/bash

export FILE_BEAT=/oldboyedu/softwares/filebeat

export PATH=$PATH:$FILE_BEAT

# 使环境变量生效

source /etc/profile.d/filebeat.sh

# 查看环境变量是否生效

which filebeat

  | filebeat参数介绍

tomcat

  | 运行第一个实例

  将标准输入的数据进行标准输出

vim stdin-to-console.yaml

filebeat.inputs:

- type: stdin

enabled: true

output.console:

pretty: true

enable: true

# 查看filebeat的输出

filebeat -e -c stdin-to-console.yaml

  企业实战

  | nginx日志收集

  安装nginx

yum -y install epel-release

yum -y install nginx

  创建配置文件

vim /etc/nginx/conf.d/elk103.oldboyedu.com.conf

server {

listen 80;



server_name es.oldboyedu.com;



root /oldboyedu/data/nginx/;



location / {

index index.html;

}

}

  创建测试数据

mkdir -p /oldboyedu/data/nginx/
echo "<h1>老男孩教育</h1>" > /oldboyedu/data/nginx/index.html

  检查配置文件

nginx -t

  启动nginx服务

systemctl start nginx

  测试nginx服务

# 编写脚本

vim /server/scripts/nginx.sh

#!/bin/bash

while true

do

for i in "curl es.oldboyedu.com"

do

Time=$((RANDOM%5 +1 ))

echo "本次间隔时间为:$Time"

curl elk103.oldboyedu.com

sleep $Time

done

done

  配置nginx收集JSON并重启nginx

# 修改nginx的配置文件

vim /etc/nginx/nginx.conf

...

# 自定义nginx的日志格式为json格式

log_format oldboyedu_nginx_json '{"@timestamp":"$time_iso8601",'

'"host":"$server_addr",'

'"clientip":"$remote_addr",'

'"size":$body_bytes_sent,'

'"responsetime":$request_time,'

'"upstreamtime":"$upstream_response_time",'

'"upstreamhost":"$upstream_addr",'

'"http_host":"$host",'

'"uri":"$uri",'

'"domain":"$host",'

'"xff":"$http_x_forwarded_for",'

'"referer":"$http_referer",'

'"tcp_xff":"$proxy_protocol_addr",'

'"http_user_agent":"$http_user_agent",'

'"status":"$status"}';





access_log /var/log/nginx/access.log oldboyedu_nginx_json;

# 测试配置文件是否正常

nginx -t

# 重新加载nginx

systemctl restart nginx

  配置filebeat的配置文件

vim 01-nginx-to-es.yaml

filebeat.inputs:

- type: log

paths:

- /var/log/nginx/access.log

tags: "nginx"

# 默认值为false,我们需要修改为true,即不会将消息存储至message字段!

json.keys_under_root: true

output.elasticsearch:

hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"]

#index: "oldboy-2022-%{[agent.version]}-%{+yyyy.MM.dd}"

indices:

- index: "oldboyedu-nginx2022-%{+yyyy.MM.dd}"

when.contains:

tags: "nginx"

# 禁用索引的生命周期!

setup.ilm.enabled: false

# 指定索引模板的名称

setup.template.name: "oldboyedu"

# 指定索引模板的匹配模式

setup.template.pattern: "oldboyedu-nginx*"

# 指定索引模板的分片信息

setup.template.settings:

index.number_of_shards: 5

index.number_of_replicas: 0

  收集nginx的错误日志

vim 02-nginx-to-es.yaml 

filebeat.inputs:

- type: log

paths:

- /var/log/nginx/access.log

tags: "nginx-access"

# 默认值为false,我们需要修改为true,即不会将消息存储至message字段!

json.keys_under_root: true





- type: log

paths:

- /var/log/nginx/error.log

tags: "nginx-error"





output.elasticsearch:

hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"]

#index: "oldboy-2022-%{[agent.version]}-%{+yyyy.MM.dd}"

indices:

- index: "oldboyedu-nginx-access-%{+yyyy.MM.dd}"

when.contains:

tags: "nginx-access"



- index: "oldboyedu-nginx-error-%{+yyyy.MM.dd}"

when.contains:

tags: "nginx-error"

# 禁用索引的生命周期!

setup.ilm.enabled: false

# 指定索引模板的名称

setup.template.name: "oldboyedu"

# 指定索引模板的匹配模式

setup.template.pattern: "oldboyedu-nginx*"

# 指定索引模板的分片信息

setup.template.settings:

index.number_of_shards: 5

index.number_of_replicas: 0

  | Nginx多虚拟主机

  配置nginx的多虚拟主机

vim /etc/nginx/conf.d/bbs.oldboyedu.com.conf

server {

listen 80;



server_name bbs.oldboyedu.com;



root /oldboyedu/data/nginx/bbs;



# 指定access.log的存储路径及日志格式.

access_log /var/log/nginx/bbs.log oldboyedu_nginx_json;



location / {

index index.html;

}

}

vim /etc/nginx/conf.d/blog.oldboyedu.com.conf

server {

listen 80;



server_name blog.oldboyedu.com;



root /oldboyedu/data/nginx/blog;



# 指定access.log的存储路径及日志格式.

access_log /var/log/nginx/blog.log oldboyedu_nginx_json;



location / {

index index.html;

}

}

  创建测试数据

mkdir -p /oldboyedu/data/nginx/{blog,bbs}

echo "<h1>blog</h1>" > /oldboyedu/data/nginx/blog/index.html

echo "<h1>bbs</h1>" > /oldboyedu/data/nginx/bbs/index.html

# 检查配置文件的语法

nginx -t

# 修改主机名映射

vim /etc/hosts

...

192.168.56.132 blog.oldboyedu.com

192.168.56.132 bbs.oldboyedu.com

# 重启nginx服务

systemctl restart nginx

# 测试服务

curl blog.oldboyedu.com

curl bbs.oldboyedu.com

  编写fielbeat的yaml

vim nginx_vm_host.yaml

filebeat.inputs:

- type: log

enabled: true

paths:

- /var/log/nginx/access.log

# false会将json解析的格式存储至message,改为true则不存储至message

json.keys_under_root: true

# 覆盖默认的message字段,使用自定义json格式的key

json.overwrite_keys: true

# 为访问日志("access.log")打标签

tags: ["nginx-access"]



- type: log

enabled: true

paths:

- /var/log/nginx/blog.log

# false会将json解析的格式存储至message,改为true则不存储至message

json.keys_under_root: true

# 覆盖默认的message字段,使用自定义json格式的key

json.overwrite_keys: true

# 为访问日志("access.log")打标签

tags: ["nginx-blog"]





- type: log

enabled: true

paths:

- /var/log/nginx/demo.log

# false会将json解析的格式存储至message,改为true则不存储至message

json.keys_under_root: true

# 覆盖默认的message字段,使用自定义json格式的key

json.overwrite_keys: true

# 为访问日志("access.log")打标签

tags: ["nginx-demo"]



- type: log

enable: true

paths:

- /var/log/nginx/error.log

# 为错误日志("error.log")打标签

tags: ["nginx-error"]



output.elasticsearch:

hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"] # index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}"

# 注意哈,下面的标签不再是"index"啦~

indices:

- index: "nginx-access-%{[agent.version]}-%{+yyyy.MM.dd}"

when.contains:

tags: "nginx-access"



- index: "nginx-error-%{[agent.version]}-%{+yyyy.MM.dd}"

when.contains:

tags: "nginx-error"



- index: "nginx-blog-%{[agent.version]}-%{+yyyy.MM.dd}"

when.contains:

tags: "nginx-blog"



- index: "nginx-demo-%{[agent.version]}-%{+yyyy.MM.dd}"

when.contains:

tags: "nginx-demo"

setup.ilm.enabled: false

# 定义模板名称.

setup.template.name: "nginx"

# 定义模板的匹配索引名称.

setup.template.pattern: "nginx-*"

[root@oldboy-es03 project]# filebeat -e -c nginx_vm_host.yaml

  | Tomcat日志收集

  部署tomcat

tar zxf apache-tomcat-10.0.6.tar.gz -C /oldboy/softwares/

cd /oldboyedu/softwares/

ln -s apache-tomcat-10.0.6 tomcat

# 配置JDK 的环境变量

vim /etc/profile.d/tomcat.sh

#!/bin/bash

export TOMCAT_HOME=/oldboyedu/softwares/tomcat

export PATH=$PATH:$TOMCAT_HOME/bin

# 让环境变量生效

. /etc/profile.d/tomcat.sh

catalina.sh

# 配置tomcat的JSON格式

vim /oldboyedu/softwares/tomcat/conf/server.xml

···(大概在133行哟~)

<Host name="tomcat.oldboyedu.com" appBase="webapps"

unpackWARs="true" autoDeploy="true">

...(需要手动注释一下原内容)

<!--

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

prefix="localhost_access_log" suffix=".txt"

pattern="%h %l %u %t "%r" %s %b" />

-->


<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"

prefix="tomcat.oldboyedu.com_access_log" suffix=".txt"

pattern="{"clientip":"%h","ClientUser":"%l","authentica

ted":"%u","AccessTime":"%t","request":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","AgentVersion":"%{User-Agent}i"}"/>

...

# 配置主机解析

vim /etc/hosts

...

19.168.56.132 tomcat.oldboyedu.com

# 启动tomcat服务

catalina.sh start

# 验证服务

略。

  使用filebeat收集日志

vim ~/conf/project/tomcat01.tomcat-to-es.yaml

filebeat.inputs:

- type: log

paths:

- /oldboyedu/softwares/tomcat/logs/tomcat.oldboyedu.com_access_log.*.txt

# false会将json解析的格式存储至message,改为true则不存储至message

json.keys_under_root: true

# 为访问日志("access.log")打标签

tags: "tomcat-access"

output.elasticsearch:

hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"]

# 注意哈,下面的标签不再是"index"啦~

indices:

- index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"

when.contains:

tags: "tomcat-access"

setup.ilm.enabled: false

# 定义模板名称.

setup.template.name: "tomcat"

# 定义模板的匹配索引名称.

setup.template.pattern: "tomcat-*"

# 指定索引模板的分片信息

setup.template.settings:

index.number_of_shards: 3

index.number_of_replicas: 0

[root@oldboy-es03 ~]#

  收集错误日志

vim  ~/conf/project/tomcat/03.tomcat-to-es.yaml

filebeat.inputs:

- type: log

paths:

- /oldboyedu/softwares/tomcat/logs/tomcat.oldboyedu.com_access_log.*.txt

json.keys_under_root: true

tags: "tomcat-access"

- type: log

paths:

- /oldboyedu/softwares/tomcat/logs/catalina*

tags: "tomcat-error"

multiline.type: pattern

multiline.pattern: '^d{2}'

multiline.negate: true

multiline.match: after

multiline.max_lines: 1000



output.elasticsearch:

hosts: ["192.168.56.130:9200","192.168.56.131:9200","192.168.56.132:9200"]

indices:

- index: "tomcat-access-%{[agent.version]}-%{+yyyy.MM.dd}"

when.contains:

tags: "tomcat-access"



- index: "tomcat-error-%{[agent.version]}-%{+yyyy.MM.dd}"

when.contains:

tags: "tomcat-error"

setup.ilm.enabled: false

setup.template.name: "tomcat"

setup.template.pattern: "tomcat-*"

setup.template.settings:

index.number_of_shards: 3

index.number_of_replicas: 0

加载全部内容

相关教程
猜你喜欢
用户评论
快盘暂不提供评论功能!