flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: Can we treat a whole file as a Flume event?
Date Tue, 22 Jan 2013 07:37:17 GMT
you cant configure it to send the entire file in an event unless you have
fixed number of events in each of the files. basically it reads the entire
file into a channel and then starts writing.

so as long as you can limit the events in the file, i think you can send
entire file as a transaction but not as a single event
as long as I understand flume treats individual lines in the file as event

if you want to pull the entire file then you may want to implement that
with messaging queues where you send an event to activemq queue and then
your consumer may pull the file in one transaction with some other
mechanism like ftp or scp or something like that

others will have better idea, i am just suggesting a crude way to get the
entire file as a single event

On Tue, Jan 22, 2013 at 12:19 PM, Henry Ma <henry.ma.1986@gmail.com> wrote:

> As far as I know, Directory Spooling Source will send the file line by
> line as an event, and File Roll Sink will receive these lines and roll up
> to a big file by a fixed interval. Is it right, and can we config it to
> send the whole file as an event?
> On Tue, Jan 22, 2013 at 1:22 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>> why don't you use directory spooling ?
>> On Tue, Jan 22, 2013 at 7:15 AM, Henry Ma <henry.ma.1986@gmail.com>wrote:
>>> Hi,
>>> When using Flume to collect log files, we want to just COPY the original
>>> files from several servers to a central storage (unix file system), not to
>>> roll up to a big file. Because we must record some messages of the original
>>> file such as name, host, path, timestamp, etc. Besides, we want to
>>> guarantee total reliability: no file miss, no file reduplicated.
>>> It seems that, in Source, we must put a whole file (size may be between
>>> 100KB and 100MB) into a Flume event; and in Sink, we must write each event
>>> to a single file.
>>> Is it practicable? Thanks!
>>> --
>>> Best Regards,
>>> Henry Ma
>> --
>> Nitin Pawar
> --
> Best Regards,
> Henry Ma

Nitin Pawar

View raw message