johnhamelink
10/15/2014 - 10:28 PM
index.html
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>fluentd</title>
<!-- jQuery from the Google CDN. -->
<script src="http://ajax.googleapis.com/ajax/libs/jquery/2.0.2/jquery.min.js"></script>
<!-- Bootstrap CSS & JS from the Bootstrap CDN. -->
<link href="http://netdna.bootstrapcdn.com/twitter-bootstrap/2.3.2/css/bootstrap-combined.min.css" rel="stylesheet" >
<script src="http://netdna.bootstrapcdn.com/twitter-bootstrap/2.3.2/js/bootstrap.min.js"></script>
<style type="text/css">
body {
margin: 2.5em auto 10em auto;
width: 50em;
max-width: 90%;
}
h2 { margin-top: 1.5em; }
h3 { margin-top: 1.25em; }
h4 { margin-top: 1em; }
h5 { margin-top: 0.75em; }
h6 { margin-top: 0.5em; }
hr { margin: 5em 0; }
#footer {
margin-top: 5em;
opacity: 0.5;
line-height: 2.5em;
}
#footer:hover { opacity: 1.0; }
</style>
</head>
<body>
<div id="content">
<h1>fluentd</h1>
<h3>Basics</h3>
<p>Fluentd is a log aggregation daemon, that allows us to collect logs from all the servers for both debugging/audit purposes, but also for performance, analysis and other cool reasons.</p>
<h3>Compatability</h3>
<p>Fluentd has a whole host of “plugins” which are used to create inputs and outputs for Fluentd. You can use fluentd to output to a mongoDB server, or Hadoop, or elasticsearch!</p>
<h3>Setup</h3>
<h4>Elasticsearch</h4>
<h5>Install Java</h5>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo apt-get update
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo apt-get install openjdk-7-jre-headless --yes
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>java -version
</code></pre>
<p>The output should contain “1.7.0”</p>
<h5>Install</h5>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>cat "deb http://packages.elasticsearch.org/elasticsearch/1.4/debian stable main" > /etc/apt/sources.list.d/elasticsearch.list
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo apt-get update && sudo apt-get install elasticsearch
</code></pre>
<h5>Configure</h5>
<p>Disable dynamic scripting as we’re going to be running kibana on the public internet:</p>
<p><strong><code>/etc/elasticsearch/elasticsearch.yml</code>:</strong></p>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>script.disable_dynamic: true
</code></pre>
<h5>Start</h5>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo service elasticsearch start
</code></pre>
<h4>Kibana</h4>
<h5>Install</h5>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>curl -L https://download.elasticsearch.org/kibana/kibana/kibana-4.0.0-BETA1.1.tar.gz | tar xzf -
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo cp -r kibana-4.0.0-BETA1.1 /usr/share
</code></pre>
<h5>Configure</h5>
<p>Since Kibana will use port 80 to talk to elasticsearch as opposed to the default 9200, kibana’s config.js must be updated.</p>
<p>Open <code>/usr/share/kibana-4.0.0-BETA1.1/config/kibana.yaml</code> and replace</p>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>port: 5601
</code></pre>
<p><strong>with:</strong></p>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>port: 80
</code></pre>
<h3>Fluentd</h3>
<h4>Installing</h4>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo apt-get update && sudo apt-get install ntp
</code></pre>
<h4>Plugins</h4>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo apt-get install make libcurl4-gnutls-dev --yes
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
</code></pre>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer
</code></pre>
<p>You can find more plugins by searching for <code>fluent-plugin</code> on rubygems.org & github.</p>
<h4>Configuring</h4>
<h5>Fluentd</h5>
<p>Next we configure Fluentd to listen to syslog messages and send them to elasticsearch.</p>
<p>Add this XML to the top of <code>/etc/td-agent/td-agent.conf</code>:</p>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code><source>
type syslog
port 5140
tag system
</source>
<match system.*.*>
type record_reformer
tag elasticsearch
facility ${tag_parts[1]}
severity ${tag_parts[2]}
</match>
<match elasticsearch>
type copy
<store>
type stdout
</store>
<store>
type elasticsearch
logstash_format true
flush_interval 5s #debug
</store>
</match>
</code></pre>
<h5>Rsyslog</h5>
<p><strong><code>/etc/rsyslog.conf</code>:</strong></p>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>*.* @127.0.0.1:5140
</code></pre>
<h4>Start</h4>
<!-- An innocent comment to force Markdown out of list parsing mode. See also http://meta.stackoverflow.com/a/99637 -->
<pre><code>sudo service td-agent start }}
</code></pre>
</div>
<div id="footer">
Exported by <a href="http://peterodding.com/code/vim/notes/" class="btn">vim-notes 0.27</a>
on Wednesday October 15, 2014 at 23:27 based on the note <code>~/Public/notes/fluentd</code>.
</div>
</body>
</html>