logstash grok 自测实例+常用正则（grok |
您所在的位置：网站首页 › partner语法 › logstash grok 自测实例+常用正则（grok |
logstash grok 自测实例+常用正则（grok
|
一、背景
研究了grok一下整天，虽然知识还是很浅薄，但还是在这里做个总结。 场景 在使用logstash进行日志收集工作的时候，filter是个很重要的插件，而其中的Grok能很好的解析日志。 logstash教程：https://blog.csdn.net/qq_34646817/article/details/81232083 grok教程：https://blog.csdn.net/qq_34646817/article/details/81232121#t1 基础知识 grok作为解析日志的插件，内含120种模式，即配置的正则。所以学习之前，不妨先去学习下正则表达式。 二、自测示例下一章罗列了常用的一些正则，规则详情也有；在进行记忆学习前，我先展示下我自己的自测学习示例。 日志我这边有一个log4j的日志，部分展示如下 [2018-07-10T16:22:30,001] AbstractBeanFactory(doGetBean:248): Returning cached instance of singleton bean 'mainCtl' [2018-07-10T16:22:30,007] MainCtl(doControl:52): current tradecode:trigger_timer,flowsn:1 [2018-07-10T16:22:30,008] FlowParser(getFlowInfo:462): debugow:C:\Users\61661\Downloads\tinyWebDemo\tinyWebDemo1\src\main\resources\flow\trigger_timer.flow [2018-07-10T16:22:30,009] MainCtl(doControl:60): flowInfos:{1=id:1,name:script} [2018-07-10T16:22:30,011] AbstractBeanFactory(doGetBean:248): Returning cached instance of singleton bean 'enumTools' [2018-07-10T16:22:30,013] AbstractBeanFactory(doGetBean:248): Returning cached instance of singleton bean 'jsDemo' [2018-07-10T16:22:30,013] JsDemo(init:18): js demo init [2018-07-10T16:22:30,019] TinyLog(debug:17): [null-trigger_timer] do function trigger_timer_1 [2018-07-10T16:22:30,019] TinyLog(info:21): [null-trigger_timer] start timer...... [2018-07-10T16:22:30,020] MainCtl(doControl:103): :2ms grok配置grok的编写配置虽然不难，但是很容易出错，要善于学会使用grok debugger来测试。 这边给一个中文版的测试网站（英文的要翻墙）：http://grok.qiexun.net/ 测试结果如下图，表达式为： \[%{TIMESTAMP_ISO8601:logdate}\] %{WORD:AA}\(%{WORD:BB}:%{WORD:BB}\): %{GREEDYDATA:C}说明：以下命令根据自己的路径进行调整 当前目录下编写conf文件 input { file { #日志所在目录 path => ["/opt/data/test.log"] type => "system" #从文件开始的地方读 start_position => "beginning" #这个是我爬过的坑 T.T ，写上下面的这段，反复重启的时候，才会从文件开始的地方读 sincedb_path => "/dev/null" } } filter { grok { match => { "message" => "\[%{TIMESTAMP_ISO8601:logdate}\] %{WORD:AA}\(%{WORD:BB}:%{WORD:BB}\): %{GREEDYDATA:C}" } } date { match => [ "logdate", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } output{ stdout{ codec=>rubydebug{} } } logstash 解析执行命令 /opt/elk/logstash-6.2.2/bin/logstash -f test.conf最后结果如下（截取了一部分）： { "host" => "hadoop02", "message" => "[2018-07-10T16:22:30,019] TinyLog(debug:17): [null-trigger_timer] do function trigger_timer_1", "@version" => "1", "logdate" => "2018-07-10T16:22:30,019", "AA" => "TinyLog", "tags" => [ [0] "_dateparsefailure" ], "C" => "[null-trigger_timer] do function trigger_timer_1", "BB" => [ [0] "debug", [1] "17" ], "path" => "/opt/data/test.log", "@timestamp" => 2018-08-01T16:43:18.303Z, "type" => "system" } { "host" => "hadoop02", "message" => "[2018-07-10T16:22:30,019] TinyLog(info:21): [null-trigger_timer] start timer......", "@version" => "1", "logdate" => "2018-07-10T16:22:30,019", "AA" => "TinyLog", "tags" => [ [0] "_dateparsefailure" ], "C" => "[null-trigger_timer] start timer......", "BB" => [ [0] "info", [1] "21" ], "path" => "/opt/data/test.log", ▽ "@timestamp" => 2018-08-01T16:43:18.303Z, "type" => "system" } { "host" => "hadoop02", "message" => "[2018-07-10T16:22:30,020] MainCtl(doControl:103): :2ms", "@version" => "1", "logdate" => "2018-07-10T16:22:30,020", "AA" => "MainCtl", "tags" => [ [0] "_dateparsefailure" ], "C" => ":2ms", "BB" => [ [0] "doControl", [1] "103" ], "path" => "/opt/data/test.log", "@timestamp" => 2018-08-01T16:43:18.303Z, "type" => "system" } 三、常用的正则说明：左边是名称，右边的是正则，意义都有在注解里面说明 当然，你自己测一下就知道了。 USERNAME [a-zA-Z0-9._-]+ USER %{USERNAME} INT (?:[+-]?(?:[0-9]+)) BASE10NUM (?[0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))) NUMBER (?:%{BASE10NUM}) BASE16NUM (?[0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+)) BASE16FLOAT \b(?[0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b POSINT \b(?:[1-9][0-9]*)\b NONNEGINT \b(?:[0-9]+)\b WORD \b\w+\b NOTSPACE \S+ SPACE \s* DATA .*? GREEDYDATA .* QUOTEDSTRING (?>(?\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``)) UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12} # Networking MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC}) CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4}) WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2}) COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2}) IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)? IPV4 (?[0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9]) IP (?:%{IPV6}|%{IPV4}) HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b) HOST %{HOSTNAME} IPORHOST (?:%{HOSTNAME}|%{IP}) HOSTPORT %{IPORHOST}:%{POSINT} # paths PATH (?:%{UNIXPATH}|%{WINPATH}) UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+ TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+)) WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+ URIPROTO [A-Za-z]+(\+[A-Za-z+]+)? URIHOST %{IPORHOST}(?::%{POSINT:port})? # uripath comes loosely from RFC1738, but mostly from what Firefox # doesn't turn into %XX URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+ #URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)? URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]* URIPATHPARAM %{URIPATH}(?:%{URIPARAM})? URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})? # Months: January, Feb, 3, 03, 12, December MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b MONTHNUM (?:0?[1-9]|1[0-2]) MONTHNUM2 (?:0[1-9]|1[0-2]) MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) # Days: Monday, Tue, Thu, etc... DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?) # Years? YEAR (?>\d\d){1,2} HOUR (?:2[0123]|[01]?[0-9]) MINUTE (?:[0-5][0-9]) # '60' is a leap second in most time standards and thus is valid. SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?) TIME (?! |
【本文地址】