flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shekhar sharma <shekhar2...@gmail.com>
Subject Re: Flume NG - preferred alternative to TailSource
Date Wed, 18 Jul 2012 06:47:52 GMT
Using tail command will not be useful if something goes wrong since it does
not maintain the state..
Better to implemet your own parser or application that reads the file and
sends the event to the flume over RPC client.

Regards,
Som

On Wed, Jul 18, 2012 at 1:17 AM, Mohammad Tariq <dontariq@gmail.com> wrote:

> Hi Jon,
>
>          Flume-NG provides an exec source that executes a given
> command and consumes the output. It should serve the purpose.
>
> Regards,
>     Mohammad Tariq
>
>
> On Tue, Jul 17, 2012 at 10:52 PM, Jon Bender <jonathan.bender@gmail.com>
> wrote:
> > Hi folks,
> >
> > I'm contemplating moving our current Flume 0.94 pipeline to Flume NG.  At
> > the moment we make heavy use of the "multitail" source with some custom
> > plugins to open up files/archives as they arrive in a specified
> directory,
> > do some parsing and then emit the parsed events.
> >
> > Ideally I would replace this stage with some Flume NG counterpart.  Is
> there
> > any Flume NG source that is designed to emit events from files, or is the
> > suggestion that we build standalone parsers / tailers and
> transmit/receive
> > the events over some standard RPC (using Avro, for instance)?
> >
> > Best,
> > Jon
>

Mime
View raw message