incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjie Lai <mjla...@gmail.com>
Subject Re: E2E mode with decorators
Date Tue, 16 Aug 2011 18:44:07 GMT
 > If I have a Decorator running on the
 > Collector that filters out events (or transforms them or something),
 > does that mean those events won't get ACK'd and thus will delivery will
 > be retried for them indefinitely?

Ack of e2e mode should work regardless any decorator configured at 
collector side. You can verify by a simple example.

[flume localhost:35873:45678] getconfigs
NODE            FLOW            SOURCE                  SINK 

collector1      default-flow    collectorSource(35853)  collector(10000) 
{regex("(.*)\\t(.*)\\t(.*)\\t", 1, "ip")  console}
agent1          default-flow    tail("/tmp/full.log") 
agentE2ESink("localhost", 35853)

I didn't get a chance to look at the source code, but I think the ack of 
e2e should be passed thru the whole collector path as an attribute. And 
it won't get affected by any decorator (unless you modify the attribute 
by intention).

Correct me if my observation is wrong.

-mingjie

On 08/16/2011 07:52 AM, Joe Crobak wrote:
> According to the Flume FAQ [1], Flume ack's events from the
> CollectorSink in E2E mode.  If I have a Decorator running on the
> Collector that filters out events (or transforms them or something),
> does that mean those events won't get ACK'd and thus will delivery will
> be retried for them indefinitely? IOW, is E2E mode unsupported in this
> situation -- or maybe is there a way for me to ACK events that I want to
> filter from the Decorator itself?
>
> Thanks,
> Joe
>
>
> [1] https://github.com/cloudera/flume/wiki/FAQ

Mime
View raw message