2013年5月9日木曜日

Python版AWSコマンドラインツールでRDSのログ(すべてのファイル)をダウンロード

スズキです。

下記でRDSのログ出力とPython版AWSコマンドラインツールを使ったログの
ダウンロード(個別)を紹介しました。
今回は個別ではなく、特定のRDSで出力されているすべてのログ(ファイル)を
ダウンロードしてみました。
("describe-db-log-files"でログファイル一覧を取得しています)

スクリプトは下記のような感じです。
# cat download-db-logs
#!/bin/sh

FILES=`aws rds describe-db-log-files \
--db-instance-identifier $1 \
| jq '.DescribeDBLogFiles[].LogFileName' \
| sed 's/"//g'`

for FILE in $FILES; do
    echo "######## $FILE ########"
    aws rds download-db-log-file-portion \
    --db-instance-identifier $1 \
    --log-file-name $FILE \
    | jq '.LogFileData' \
    | sed 's/^"//' \
    | sed 's/"$//' \
    | sed 's/\\t/\t/g' \
    | sed 's/\\n/\n/g'
done

実行すると次のような感じになります。
# ./download-db-logs master
######## error/mysql-error-running.log ########
null
######## error/mysql-error-running.log.12 ########
130502 11:11:27 [Warning] IP address '10.0.0.210' could not be resolved: Name or service not known
...
######## error/mysql-error-running.log.9 ########
130502  8:02:03 [Note] /rdsdbbin/mysql/bin/mysqld: Normal shutdown
...
######## error/mysql-error.log ########
null
######## general/mysql-general.log ########
/rdsdbbin/mysql/bin/mysqld, Version: 5.5.27-log (Source distribution). started with:
...
######## general/mysql-general.log.12 ########
/rdsdbbin/mysql/bin/mysqld, Version: 5.5.27-log (Source distribution). started with:
...
######## general/mysql-general.log.13 ########
/rdsdbbin/mysql/bin/mysqld, Version: 5.5.27-log (Source distribution). started with:
...
######## general/mysql-general.log.14 ########
/rdsdbbin/mysql/bin/mysqld, Version: 5.5.27-log (Source distribution). started with:
...
######## slowquery/mysql-slowquery.log ########
null
######## slowquery/mysql-slowquery.log.12 ########
/rdsdbbin/mysql/bin/mysqld, Version: 5.5.27-log (Source distribution). started with:
...
######## slowquery/mysql-slowquery.log.13 ########
/rdsdbbin/mysql/bin/mysqld, Version: 5.5.27-log (Source distribution). started with:
...
######## slowquery/mysql-slowquery.log.14 ########
/rdsdbbin/mysql/bin/mysqld, Version: 5.5.27-log (Source distribution). started with:
...

ただし、"Marker"や"AdditionalDataPending"は考慮していません。
"AdditionalDataPending: true"、つまり未ダウンロードのログが残っている場合は、
そのログはダウンロードできません...

そろそろシェルスクリプトの限界か...これ以上はRubyで書くか...
--------
http://www.suz-lab.com

7 コメント:

匿名 さんのコメント...

Simply wish to say your article is as astonishing. The clarity in your post is just great and i can assume you're an expert on this subject. Well with your permission let me to grab your RSS feed to keep updated with forthcoming post. Thanks a million and please continue the enjoyable work.

Here is my blog - xerox 8560

匿名 さんのコメント...

If some one needs to be updated with newest technologies therefore
he must be pay a quick visit this website and be up to date everyday.


my web-site :: great retriever Training forum information

匿名 さんのコメント...

Thanks for the marvelous posting! I seriously enjoyed reading it, you
happen to be a great author. I will be sure to bookmark your
blog and definitely will come back very soon.
I want to encourage you to continue your great posts, have a nice evening!


My webpage :: excellent breed golden retrievers stuff

匿名 さんのコメント...

Most of what you assert is supprisingly appropriate and that
makes me wonder why I hadn't looked at this in this light before. Your article truly did turn the light on for me personally as far as this subject matter goes. Nevertheless at this time there is actually 1 issue I am not too comfortable with so while I make an effort to reconcile that with the central theme of your position, allow me see what all the rest of your visitors have to point out.Very well done.

Look at my weblog ... picamilon

匿名 さんのコメント...

Hi, i think that i noticed you visited my site thus i
got here to return the favor?.I'm attempting to find issues to improve my site!I guess its ok to make use of some of your ideas!!

Here is my weblog; Internet Money Path Reviews

匿名 さんのコメント...

Greetings! I've been following your website for some time now and finally got the courage to go ahead and give you a shout out from Huffman Texas! Just wanted to mention keep up the excellent work!

Here is my web page: Test Force Xtreme

匿名 さんのコメント...

My brother suggested I might like this blog.
He was totally right. This post actually made my day. You can not imagine just
how much time I had spent for this info! Thanks!


Also visit my web site; Art::Photography