flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ychu" <y...@isoftstone.com>
Subject Re: tailDir
Date Fri, 18 May 2012 01:51:39 GMT
tailDir("dirname"[, fileregex=".*"[, startFromEnd=false[, recurseDepth=0]]]{,delim="regex",
delimMode="exclude|prev|next"}) :: 

you should try 

tail("filename"[, startFromEnd=false]{,delim="regex", delimMode="exclude|prev|next"}) :: Similar
to Unix’s tail utility. One line is one event. Generates events for the entire file then
stays open for more data, and follows filename. (e.g. if tailing file "foo" and then "foo"
is moved to "bar" and a new file appears named "foo", it will finish reading the new "bar"
file and then start from the beginning of "foo"). If the startFromEnd parameter is false,
tail will re-read from the beginning of the file. If it is true, it will only start reading
from the current end of file. If the last line of a file does not end with a newline character
(\n), the tail source will only send an event with this last line when the tail is closed.
See the section on tailing a file for details on delim and delimMode.

The tail source by default assumes \n as a delimiter, and excludes the delimiter from events.
There are optional line delimiter arguments that allow you to specify arbitrary regular expressions
as delimiters and to specify if the delimiter should be part of the prev ious event, next
event, or exclude d.
Here are some examples and scenarios to illustrate:
The following example tails a file that requires two or more consecutive new lines to be considered
a delimiter. The newlines are excluded from the events.
tail("file", delim="\n\n+", delimMode="exclude")
This example tails a file and uses </a> as a delimiter, and appends the delimiter to
the previous event. This could serve as a quick-and-dirty xml record splitter.
tail("file", delim="</a>", delimMode="prev")
Finally, this example tails a file and uses the regex "\n\d\d\d\d" as a delimiter and appends
the delimiter to the next event. This could be used to gather lines from a stack dump in a
log file that starts with four digits (like a year from a date stamp).
tail("file", delim="\\n\\d\\d\\d\\d", delimMode="next")



发件人: M@he$h 
发送时间: 2012-05-18  09:32:24 
收件人: flume-user@incubator.apache.org 
主题: tailDir 

We have a directory where there will be log files for each day being written. for example


I want flume to tail files depending on that day's date. for example : as today's date is
05/17/2012 , it should only tail file names which have 05172012 at the end.

Can we do it using tailDir? or can you suggest the best way to do this in flume?

Look forward for your response.

Thanks and Regards,

View raw message