1. 홈
  2. 질의 응답
  3. Fluentd가 amazon s3에 로그를 쓰지 않았습니다.

Fluentd가 amazon s3에 로그를 쓰지 않았습니다.

2016-09-23 2 views
0

fluentd-s3-plugin을 테스트하려고하는데 아직 로그를 s3 버킷에 게시하지 않습니다. 나는 td-agent로 유창하게 설치 한 우분투 xenial에서 모든 것을 돌리고있다. 다음 여기에 또한 /var/log/td-agent/td-agent.logFluentd가 amazon s3에 로그를 쓰지 않았습니다.

2016-09-23 09:16:18 -0300 [info]: reading config file path="/etc/td-agent/td-agent.conf" 
2016-09-23 09:16:18 -0300 [info]: starting fluentd-0.12.26 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-kinesis' version '1.1.1' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-mongo' version '0.7.13' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-s3' version '0.6.8' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-scribe' version '0.10.14' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td' version '0.10.28' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluentd' version '0.12.26' 
2016-09-23 09:16:18 -0300 [info]: adding match pattern="nginx.log" type="s3" 
2016-09-23 09:16:18 -0300 [info]: adding source type="tail" 
2016-09-23 09:16:18 -0300 [info]: using configuration file: <ROOT> 
    <source> 
    type tail 
    format nginx 
    path /var/log/nginx/access.log 
    pos_file /var/log/td-agent/nginx-access.pos 
    tag nginx.log 
    </source> 
    <match nginx.log> 
    @type s3 
    s3_bucket kd.creatives 
    aws_key_id xxxxxx 
    aws_sec_key xxxxxx 
    s3_region us-west-2 
    path logs/ 
    buffer_path /var/log/td-agent/s3 
    time_slice_format %Y%m%d%H%M 
    utc 
    format_json true 
    include_time_key true 
    buffer_chunk_limit 256m 
    </match> 
</ROOT> 
2016-09-23 09:16:18 -0300 [warn]: parameter 'format_json' in <match nginx.log> 
    @type s3 
    s3_bucket kd.creatives 
    aws_key_id xxxxxx 
    aws_sec_key xxxxxx 
    s3_region us-west-2 
    path logs/ 
    buffer_path /var/log/td-agent/s3 
    time_slice_format %Y%m%d%H%M 
    utc 
    format_json true 
    include_time_key true 
    buffer_chunk_limit 256m 
</match> is not used. 
2016-09-23 09:16:20 -0300 [info]: following tail of /var/log/nginx/access.log 
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""

내 TD-agent.conf입니다 파일에있는 TD-에이전트 로그 파일입니다

# Centralized nginx logs into Amazon s3 
<source> 
    type tail 
    format nginx 
    path /var/log/nginx/access.log 
    pos_file /var/log/td-agent/nginx-access.pos 
    tag nginx.log 
</source> 

<match nginx.log> 
    @type s3 
    s3_bucket xxxxxxx 
    aws_key_id xxxxxxxxxxxxxxxx 
    aws_sec_key xxxxxxxxxxxxxxxxxxxxxxxxxx 
    s3_region us-west-2 

    path logs/ 
    buffer_path /var/log/td-agent/s3 
    time_slice_format %Y%m%d%H%M 

    utc 

    format_json true 
    include_time_key true 
    buffer_chunk_limit 256m 
</match>

fluentd가 로그를 s3 버킷에 게시하지 않는 이유에 대한 답변을 보내 주시면 감사하겠습니다.

출처

2016-09-23 francotestori

답변

1

in_tail에서 nginx 로그를 구문 분석하지 못했습니다. 그래서 fluentd는 s3에 모든 로그를 보내지 않았습니다. format nginx은 조합 로그 형식을 기본값으로 처리합니다.

사용자 정의 형식을 사용하여 nginx 로그를 구문 분석해야한다고 생각합니다.

<source> 
    type tail 
    format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwarder>[^\"]*)")?/ 
    time_format %d/%b/%Y:%H:%M:%S %z 
    path /var/log/nginx/access.log 
    pos_file /var/log/td-agent/nginx-access.pos 
    tag nginx.log 
</source>

Ref. http://docs.fluentd.org/articles/in_tail

자신의 정규식을 fluentd-ui로 테스트 할 수 있습니다. http://docs.fluentd.org/articles/fluentd-ui

출처

2016-09-24 11:34:29 torut

+0

답장을 보내 주셔서 감사합니다. 방금 fluentd로 시작했고 이제는 로그 정규식 형식 (루비와 동일)에 익숙해졌습니다. – francotestori

 관련 문제

  • 관련 문제 없음^_^