flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ychu" <y...@isoftstone.com>
Subject Re: tailDir
Date Fri, 18 May 2012 01:51:39 GMT
tailDir("dirname"[, fileregex=".*"[, startFromEnd=false[, recurseDepth=0]]]{,delim="regex",
delimMode="exclude|prev|next"}) :: 


you should try 

tailDir("/usr/local/current/logs/",fileregex="omq.log.%m%d%Y",startFromEnd=true,recurseDepth=3)
--------------------------------------------explain-----------------
tail("filename"[, startFromEnd=false]{,delim="regex", delimMode="exclude|prev|next"}) :: Similar
to Unix’s tail utility. One line is one event. Generates events for the entire file then
stays open for more data, and follows filename. (e.g. if tailing file "foo" and then "foo"
is moved to "bar" and a new file appears named "foo", it will finish reading the new "bar"
file and then start from the beginning of "foo"). If the startFromEnd parameter is false,
tail will re-read from the beginning of the file. If it is true, it will only start reading
from the current end of file. If the last line of a file does not end with a newline character
(\n), the tail source will only send an event with this last line when the tail is closed.
See the section on tailing a file for details on delim and delimMode.



The tail source by default assumes \n as a delimiter, and excludes the delimiter from events.
There are optional line delimiter arguments that allow you to specify arbitrary regular expressions
as delimiters and to specify if the delimiter should be part of the prev ious event, next
event, or exclude d.
Here are some examples and scenarios to illustrate:
The following example tails a file that requires two or more consecutive new lines to be considered
a delimiter. The newlines are excluded from the events.
tail("file", delim="\n\n+", delimMode="exclude")
This example tails a file and uses </a> as a delimiter, and appends the delimiter to
the previous event. This could serve as a quick-and-dirty xml record splitter.
tail("file", delim="</a>", delimMode="prev")
Finally, this example tails a file and uses the regex "\n\d\d\d\d" as a delimiter and appends
the delimiter to the next event. This could be used to gather lines from a stack dump in a
log file that starts with four digits (like a year from a date stamp).
tail("file", delim="\\n\\d\\d\\d\\d", delimMode="next")





2012-05-18 



  
胡媛聪
软通动力信息技术(集团)有限公司 
地址:深圳市南山区高新区高新南九道9号威新软件科技园8号楼3层
邮编:518057
电话:0755-26719999
手机:13556898820
Email:ychu@isoftstone.com



发件人: M@he$h 
发送时间: 2012-05-18  09:32:24 
收件人: flume-user@incubator.apache.org 
抄送: 
主题: tailDir 
Hello,

We have a directory where there will be log files for each day being written. for example
:

abc.log.05162012
pqr.log.05162012
abc.log.05172012
pqr.log.05172012


I want flume to tail files depending on that day's date. for example : as today's date is
05/17/2012 , it should only tail file names which have 05172012 at the end.

Can we do it using tailDir? or can you suggest the best way to do this in flume?

Look forward for your response.

-- 
Thanks and Regards,

Mahesh
619-816-7011.
Mime
View raw message