2013年1月15日火曜日

FluentdからCloudWatchのカスタムメトリクスにデータを登録してみた(UPDATE2!)

スズキです。
CloudWatchプラグインは、こちらが最新です!(本記事は少し古い...)
"dstat(fluent-plugin-dstat)"の情報をCloudWatchに登録してみた
下記を紹介したところですが、更にアップデートしてしまいました...
FluentdからCloudWatchのカスタムメトリクスにデータを登録してみた(UPDATE!)

変更点

  • DataCounterの一つの集計ログからCloudWatchへ複数のデータを登録可能に
    • DataCounterは一つでよい("patternN"で区別)
    • Dimensionをタグ(Tag)からキー(Key)に変更
      • Keyには"patternN"のキーを登録

設定ファイル(/etc/td-agent/td-agent.conf)

<source>
    type     tail
    format   syslog
    path     /opt/suz-lab/var/log/syslog/all.log
    pos_file /opt/suz-lab/var/lib/td-agent/pos/tail.syslog.pos
    tag      tail.syslog
</source>
<match tail.syslog>
    type copy
    <store>
        type file
        path /tmp/tail.syslog
    </store>
    <store>
        type      datacounter
        unit      minute
        aggregate all
        count_key ident
        pattern1  history ^-bash$
        pattern2  audit   ^audispd$
        tag       datacounter.syslog
    </store>
</match>
<match datacounter.syslog>
    type copy
    <store>
        type file
        path /tmp/datacounter.syslog
    </store>
    <store>
        type                 cloudwatch
        buffer_type          file
        buffer_path          /opt/suz-lab/var/lib/td-agent/buf/cloudwatch.datacounter.syslog
        flush_interval       1m
        cloud_watch_endpoint monitoring.ap-northeast-1.amazonaws.com
        namespace            SUZ-LAB/TEST
        <metric>
            key  history_count
            name SyslogCount
            unit Count
        </metric>
        <metric>
            key  audit_count
            name SyslogCount
            unit Count
        </metric>
    </store>
</match>

CloudWatchプラグイン(/etc/td-agent/plugin/out_cloudwatch.rb

module Fluent
    require 'aws-sdk'
    class CloudWatchOutput < TimeSlicedOutput

        MAX_METRIC_DATA_SIZE = 20 
        Fluent::Plugin.register_output('cloudwatch', self)
        config_param :aws_key_id,           :string, :default => nil
        config_param :aws_sec_key,          :string, :default => nil
        config_param :cloud_watch_endpoint, :string, :default => 'monitoring.ap-northeast-1.amazonaws.com'
        config_param :namespace,            :string

        def initialize
            super
            @metrics = []
        end

        def configure(conf)
            super
            conf.elements.select {|e|
                e.name == 'metric'
            }.each do |e|
                @metrics << {
                    'name' => e['name'],
                    'key'  => e['key'],
                    'unit' => e['unit']
                }
            end
        end

        def start
            super
            AWS.config(
                :access_key_id        => @aws_key_id,
                :secret_access_key    => @aws_sec_key,
                :cloud_watch_endpoint => @cloud_watch_endpoint
            )
        end

        def shutdown
            super
        end

        def format(tag, time, record)
            record["tag"]       =  tag
            record["timestamp"] =  Time.at(time).iso8601
            record.to_msgpack
        end
        
        def write(chunk)
            metric_data = []
            chunk.msgpack_each do |record|
                $log.debug(record.inspect)
                @metrics.each do |metric|
                    tmp_data_by_key = {
                        :metric_name => metric['name'],
                        :timestamp   => record['timestamp'],
                        :value       => record[metric['key']],
                        :unit        => metric['unit'],
                        :dimensions  => [ { :name  => 'Key', :value => metric['key'] } ]
                    }
                    metric_data << tmp_data_by_key
                    tmp_data_by_key_and_hostname = Marshal.load(Marshal.dump(tmp_data_by_key))
                    tmp_data_by_key_and_hostname[:dimensions] << {
                        :name  => 'HostName', :value => Socket.gethostname
                    }
                    metric_data << tmp_data_by_key_and_hostname
                end
            end
            $log.debug(metric_data.inspect)
            until metric_data.length <= 0 do
                tmp_data = metric_data.slice!(0, MAX_METRIC_DATA_SIZE)
                $log.debug(tmp_data.inspect)
                AWS::CloudWatch.new.put_metric_data(
                    :namespace   => @namespace,
                    :metric_data => tmp_data
                )
            end
        end

    end
end

"fluent-plugin-dstat"からCloudWatchへも、やってみよう。
--------
http://www.suz-lab.com

0 コメント: