flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Flume Jambalaya - A Flume Plugin with Multiple Components
Date Sat, 03 May 2014 00:29:18 GMT
Had a quick look.  Added my observations about FileSource stuff in
https://issues.apache.org/jira/browse/FLUME-2344

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Fri, May 2, 2014 at 8:21 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:

> What Wolfgang said :)
>
> I'd be interested in hearing how the File Source is different from or
> better than Exec Source with tail -F or
> https://issues.apache.org/jira/browse/FLUME-2344 - do you know?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, May 2, 2014 at 5:18 PM, Israel Ekpo <israel@aicer.org> wrote:
>
>> Flume Community,
>>
>> I created a Flume Plugin with multiple components that complements the
>> current version of Apache Flume.
>>
>> This was necessary as part of a personal project as I working on.
>>
>> It is code named - Flume Jambalaya
>>
>> Jambalaya is a standalone Apache Flume plugin that contains a variety of
>> sources, interceptors, channels, sinks, serializers and other components
>> designed to extend the Flume architecture. It has been released under the
>> Apache License version 2.0
>>
>> https://github.com/aicer/flume-jambalaya
>>
>> It currently contains:
>>
>> (a) File Source - This source lets you ingest data by tailing files from
>> a specific path
>> (b) ElasticSearch HTTP Sink - This sink sends events to an ElasticSearch
>> cluster via HTTP with no dependency on the ElasticSearch versions between
>> Flume and the Server cluster.
>> (c) DateInterceptor - The date interceptor is used for parsing dates from
>> fields and using that date or timestamp as the timestamp for the Flume
>> event.
>> (d) Grok Interceptor - allows you to extract structured data from
>> unstructured text and inject them as headers into the event
>>
>> Sample configuration files are available here
>>
>>
>> https://github.com/aicer/flume-jambalaya/tree/master/sample-configuration-files
>>
>> I did not realize that the Flume trunk already has a HTTP Sink for
>> ElasticSearch so you can decide whether or not to use the sink that comes
>> with it
>>
>> I am still testing and integrating the various components.
>>
>> Please check it out when you get a chance and send me some feedback
>>
>> Thanks.
>>
>
>

Mime
View raw message