flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SaravanaKumar TR <saran0081...@gmail.com>
Subject Need suggestion on reliable source for log processing
Date Mon, 27 Oct 2014 09:21:05 GMT
Hi,

I am using Apache flume 1.5.0.Quick setup explanation here.

Source:exec , tail –F command for a logfile.

Channel:  file channel

Sink: HDFS

Use case:to move real time data from logfile to HDFS.


It appears like exec is not a reliable source , as we may data loss if
channel/source is down.


So i tried with other option "spooling directory source" which is mentioned
as reliable source.But here I have a single logfile where data gets
appended in , so I dont see option of moving the file to spool directory.


Can anyone help me with providing any other reliable source option in case
where logfile gets appended with data and logfile rotation happens only at
the end of the day.


Thanks,

Saravana

Mime
View raw message