1. logstash 구성도
2. logtash configure
# The # character at the beginning of a line indicates a comment. Use # comments to describe your configuration. input { } # The filter part of this file is commented out to indicate that it is # optional. # filter { # # } output { }
input {
stdin { }
}
output {
stdout {
codec => rubydebug -- pretty display
}
}
input {
stdin { }
}
output {
elasticsearch { host = localhost } } -- elasticsearch
}
JVM등 리소스 사용량을 줄이기 위해 logstash forwarder 사용
Logstash forwarder -config Logstash forwarder.conf
input {
lumberjack {
port => 2000
ssl_certificate => "path/to/ssl.crt"
ssl_key => "path/to/ssl.key"
type => "log type"
}
}
input {
file {
path => files... (csv)
start_position => default end.. (options...)
tags => (options..)
type => (options..)
}
}
filter {
csv {
columns => colname1, colname2...
separator => default ","
}
}
filter {
date {
match => [...] match => [ "field", "MMM dd YYY HH:mm:ss", "MMddYYYY",...]
target => default : "@timestamp"
timezone =>...
}
date { -- date 타입 map
match => ["field", "yyyy-MM-dd"]
target => "@timestamp"
}
mutate {
convert => hash of field and data type
( convert => ["t1", "float"]
convert => ["t2", "float"]
convert => ["t3", "integer"]
)
join =>
lowercase =>
merge =>
rename =>
replace =>
split =>
strip =>
uppercase =>
}
}
output {
elasticsearch {
action => default ("index") or "delete"
cluster =>
host =>
document_id => default ("nil")
index => default ("logstash-%{+YYYY.MM.dd}")
index_type =>
port =>
protocol => "node", "transport", "http"
}
}
3. logstash plugins
1) input plugin
- file : stemas log events from file
- redis : from redis instance
- stdin : from standard input
- syslog : from standard input
- ganglia : ganglia packets over the network via udp.
- lumberjack : lumberjack protocol
- eventlog : from windows event log
- s3 : from s3 bucket
- elasticsearch :
input {
file {
path => "/path/logfiles/*"
add_field => { "input_time" => "%{@timestamp}" }
codec => "json" (default : "plan")
delimiter => "@" (default : "\n")
exclude => "*.gz"
sincedb_path => "/tmp/.sincedb*" -- file read 위치 정보
sincedb_write_interval => "30" (default 15 secound)
start_position => "end" ("beginning" or "end")
tags => ["test"] --> filter에서 조건을 체크 할수 있음.
}
file {
path => ["/tmp/t1/*"]
type => "syslog" --> type에 의한 filter 조건 처리
}
file {
path => ["/tmp/t2/*"]
type => "apache"
}
stdin {
add_field =>
codec =>
tags => (default : line)
type =>
}
}
filter {
if "test" in tags[] { -- tag에 의한 조건처리
}
if [type] == "syslog" {
grok {
}
}
if [type] == "apache" {
grok {
}
}
}
2) filter plugin
- date : parse data from incoming events.
- drop : drop from incoming events
- grok : parse unstructured date from events to structured format
- multiline : multiple lines to one logstash event
- dns : resolve ip address
- mutate : rename, remove, modify, replace fields
- geoip : add geographic info based on ip address.
3) output plugin
- file
- elasticsearch
- stdout
- redis
- mongodb
- kafka
4) codec plugin
- avro
- json
- line
- multiline
- plain
- rubydebug
- spool
4. plugins 확인
1) plugin 전체 리스트
bin/plugin list
2) plugin 이름으로 조회
bin/plugin list <name...>
3) plugin group name으로 조회
bin/plugin list --group input/filter/output
5. plugin 데이터 타입
1) array : path => [ "v1", "v2" ]
2) boolean : periodic_path => false
3) codec : codec => "json"
4) hash : match => { "k1" => "v1" "k2" => "v2" }
5) string : value => "test"
6) comments : #test
7) field references : [field_name]...
6. logstash 조건
1) 형식
if < conditional exp > {
}
else if < conditional exp > {
}
else {
}
2) comparison operators.
Equality operators: ==, !=, <, >, <=, >=
Regular expressions: =~, !~
Inclusion: in, not in
Boolean : and, or, nand, xor
Unary operators : !
3) example
filter {
if [action] == "test" {
mutate { remove => "data" }
}
}
output {
if [log] == "ERROR" and [level] == "CRITICAL" {
email {
}
}
}
7. plugin별 세부 옵션 사항
1) input
input {
twitter {
add_field =>
codec =>
consumer_key =>
consumer_secret =>
full_tweet => (default : false)
keywords => array type ["test1", "test2"]
oauth_token => 인증 토큰
oauth_token_secret => 인증 토큰
tags =>
type =>
}
}
input {
lumberjack {
port => (require)
ssl_certificate => (require)
ssl_key => (require)
add_field =>
codec =>
host => (default : 0.0.0.0)
ssl_key_passphrase =>
tags =>
type =>
}
}
input {
redis {
add_field =>
codec =>
data_type => "list" or "channel" or "pattern_channel"
host =>
key => "list" or "channel"
password =>
port =>
}
}
input {
jdbc {
statement => "SELECT id, mycolumn1, mycolumn2 FROM my_table WHERE id > :sql_last_value"
use_column_value => true
tracking_column => id
# ... other configuration bits
}
}
output {
csv {
path => "/tmp/t1.csv"
fields => ["col1","col2","col3"]
codec => default("plain")
csv_options => ["col_sep" => "|" "row_sep" => "\r\n"]
gzip => default :
file => "/tmp/csv.out"
max_size => ??
}
}
output {
eamil {
to => "test@test.com"
attachment => [ ] array of file
body =>""
cc => multi e-mail ids in a comma separated
from => default ("logstash.alert@nowhere.com")
to =>
htmlbody => html format
replyto =>
subject => default ("")
}
}
output {
elasticsearch {
action => "index"
bind_port =>
bind_host =>
cacert =>
cluster =>
document_id =>
document_type =>
host =>
index => default : "logstash-%{+YYYY.MM.dd}"
max_retries => default : 3
node_name =>
password =>
port =>
user =>
}
}
output {
ganglia {
metric =>
unit =>
value =>
}
}
install : bin/plugin install logstash-output-jira
output {
jira {
issuetypeid =>
password =>
priority =>
projectid =>
summary =>
username =>
assignee =>
reporter =>
}
}
output {
kafka {
topic_id =>
}
}
output {
lumberjack {
hosts =>
port =>
ssl_certificate =>
}
}
output {
rabbitmq {
exchange =>
exchange_type =>
host =>
}
}
output {
stdout { }
}
install : bin/plugin install logstash-output-mongodb
output {
mongodb {
collection =>
database =>
uri =>
}
}
filter {
mutate {
add_field => { }
add_tag => []
covert =>
join =>
lowercase =>
merge =>
remove_field =>
remove_tag =>
replace =>
split =>
strip =>
update =>
uppercase =>
}
}
input {
file {
path => "/test/test.lst"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
negate => true
what => previous
}
}
}
7. 안정적인 logstash 사용구조
