FollowKMan

Process AWS ELB logs with Logstash! update!

Posted on Apr 7, 2016

I previously wrote about making a grok filter to process ELB logs.
I have since worked on this further and developed an updated filter, which has been working very well for some time now.
As there were some comments on the previous post, I thought I should upload my working copy that I use right now also.

As it is right now, I’ve broken the filter into 4 parts.

So what does this mean?
Here it is broken down.

Here is what I’ve done!

Part1 – filters the main logs, and allows for the 4 weird log example scenarios.
-It also creates a new field named ‘elb_request’, which separates out the full request.

%{TIMESTAMP_ISO8601:timestamp}
%{NOTSPACE:elb_name}
%{IP:elb_client_ip}:%{INT:elb_client_port:int}
(?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-)
%{NUMBER:request_processing_time:float}
%{NUMBER:backend_processing_time:float}
%{NUMBER:response_processing_time:float}
(?:%{INT:elb_status_code:int}|-)
(?:%{INT:backend_status_code:int}|-)
%{INT:elb_received_bytes:int}
%{INT:elb_sent_bytes:int}
\"(?:%{GREEDYDATA:elb_request}|-)\"
\"(?:%{GREEDYDATA:userAgent}|-)\"
%{NOTSPACE:elb_sslcipher}
%{NOTSPACE:elb_sslprotocol}

Part2 – matches that ‘ELB Online’ message I’d like to keep.
-I want to keep this part to use the ‘ELB Online’ events in dashboards.

%{GREEDYDATA:event_name} for ELB:
%{NOTSPACE:elb_name} at
%{TIMESTAMP_ISO8601:timestamp}

Part3 – investigates the ‘elb_request field’ of Part1 and breaks it further into 4 new fields.
-It also creates a new field named ‘http_path’.

(?:%{WORD:http_method})
(?:%{DATA:http_path})?
(?:%{DATA:http_type}
/%{NUMBER:http_version:float})?
|%{GREEDYDATA:rawrequest}

Part4 – breaks down the new ‘http_path’ field to component parts.
-I use this for further analysing web trends.

(?:%{WORD:http_path_protocol}://)?
(%{NOTSPACE:http_path_site}:)?
(?:%{NUMBER:http_path_port:int})?
(?:%{GREEDYDATA:http_path_url})?

How does the raw code look?
Note: This is useful to understand where spaces go etc.

Part1:

%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_name} %{IP:elb_client_ip}:%{INT:elb_client_port:int} (?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{INT:elb_status_code:int}|-) (?:%{INT:backend_status_code:int}|-) %{INT:elb_received_bytes:int} %{INT:elb_sent_bytes:int} \"(?:%{GREEDYDATA:elb_request}|-)\" \"(?:%{GREEDYDATA:userAgent}|-)\" %{NOTSPACE:elb_sslcipher} %{NOTSPACE:elb_sslprotocol}

Part2:

%{GREEDYDATA:event_name} for ELB: %{NOTSPACE:elb_name} at %{TIMESTAMP_ISO8601:timestamp}

Part3:

(?:%{WORD:http_path_protocol}://)?(%{NOTSPACE:http_path_site}:)?(?:%{NUMBER:http_path_port:int})?(?:%{GREEDYDATA:http_path_url})?

Part4:

(?:%{WORD:http_path_protocol}://)?(%{NOTSPACE:http_path_site}:)?(?:%{NUMBER:http_path_port:int})?(?:%{GREEDYDATA:http_path_url})?

And how does this look as a grok filter?
NOTE: In my ‘input’ implementation, I have labelled all my incoming ELB Logs as type ‘elblogs’

SO HERE IT IS!
This is the complete filter, that you can copy/paste to test yourself if you like:

filter {
if [type] == "elblogs" {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb_name} %{IP:elb_client_ip}:%{INT:elb_client_port:int} (?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{INT:elb_status_code:int}|-) (?:%{INT:backend_status_code:int}|-) %{INT:elb_received_bytes:int} %{INT:elb_sent_bytes:int} \"(?:%{GREEDYDATA:elb_request}|-)\" \"(?:%{GREEDYDATA:userAgent}|-)\" %{NOTSPACE:elb_sslcipher} %{NOTSPACE:elb_sslprotocol}"]
match => ["message", "%{GREEDYDATA:event_name} for ELB: %{NOTSPACE:elb_name} at %{TIMESTAMP_ISO8601:timestamp}"]
}
if [elb_request] =~ /.+/ {
grok {
match => ["elb_request", "(?:%{WORD:http_method}) (?:%{DATA:http_path})? (?:%{DATA:http_type}/%{NUMBER:http_version:float})?|%{GREEDYDATA:rawrequest}"]
}
}
if [http_path] =~ /.+/ {
grok {
match => ["http_path", "(?:%{WORD:http_path_protocol}://)?(%{NOTSPACE:http_path_site}:)?(?:%{NUMBER:http_path_port:int})?(?:%{GREEDYDATA:http_path_url})?"]
}
}
geoip {
source => "elb_client_ip"
}
}
date {
match => [ "timestamp", "ISO8601" ]
}
useragent {
source => "userAgent"
prefix => "browser_"
}
}

Again, I hope this is useful to someone.

If you found it useful, then why not leave a comment! 😉