flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paweł <pro...@gmail.com>
Subject AWS S3 flume source
Date Thu, 31 Jul 2014 14:27:00 GMT
Hi,
I'm wondering if Flume is able to read directly from S3.

I'll describe my case. I have log files stored in AWS S3. I have to fetch
periodically new S3 objects and read log lines from it. Than use log lines
(events) are processed in standard flume's way (as with other sources).

*1) Is there any way to fetch S3 objects or I have to write my own Source?*

There is also second case. I want to have flume configuration dynamic.
Flume sources can change in time. New AWS key and S3 bucket can be added or
deleted.

*2) Is there any other way to configure Flume than by static configuration
file?*

--
Paweł Róg

Mime
View raw message