flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Memory channel implemented by CircularFifoQueue (purging oldest event if full)
Date Mon, 30 Mar 2015 04:05:12 GMT
Hi Herman,

Maybe this will help:
http://search-hadoop.com/m/DEeB4lfnL2/otis+oldest&subj=Re+Asynchronous+SpillableMemoryChannel

IMHO it would be nice to add this to the existing Spillable Memory Channel.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Mon, Mar 23, 2015 at 3:22 PM, Herman Schistad <herman.schistad@gmail.com>
wrote:

> Hi,
>
> Simple question: is there an implementation of a Flume
> memory/file-channel based on a Circular Fifo Queue
> (
> http://commons.apache.org/proper/commons-collections/javadocs/api-release/org/apache/commons/collections4/queue/CircularFifoQueue.html
> )
> anywhere?
>
> That is instead of rejecting new events once the memory channel is
> full, I want to purge out the oldest one(s) until I can fit my new
> incoming event.
>
> I can see from the source code that the default MemoryChannel is
> implemented using a LinkedBlockingDeque, which doesn't suit my needs
> as I (for some channels/sinks) don't care about some events
> disappearing.
>
> Appreciate thoughts, ideas and experiences with this.
>
> --
> Herman Schistad
>

Mime
View raw message