flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: AWS S3 flume source
Date Thu, 31 Jul 2014 17:26:04 GMT
In both cases, Sharninder is right :)

Sharninder wrote:
>
> As far as I know, there is no (open source) implementation of an S3
> source, so yes, you'll have to implement your own. You'll have to
> implement a Pollable source and the dev documentation has an outline
> that you can use. You can also look at the existing Execsource and
> work your way up.
>
> As far as I know, there is no way to configure flume without using the
> configuration file.
>
>
>
> On Thu, Jul 31, 2014 at 7:57 PM, Paweł <prog88@gmail.com
> <mailto:prog88@gmail.com>> wrote:
>
> Hi,
> I'm wondering if Flume is able to read directly from S3.
>
> I'll describe my case. I have log files stored in AWS S3. I have
> to fetch periodically new S3 objects and read log lines from it.
> Than use log lines (events) are processed in standard flume's way
> (as with other sources).
>
> *1) Is there any way to fetch S3 objects or I have to write my own
> Source?*
>
> There is also second case. I want to have flume configuration
> dynamic. Flume sources can change in time. New AWS key and S3
> bucket can be added or deleted.
>
> *2) Is there any other way to configure Flume than by static
> configuration file?*
>
> --
> Paweł Róg
>

Mime
View raw message