flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Shannon <cshannon...@gmail.com>
Subject Re: Flume Configuration & topology approach
Date Mon, 07 Apr 2014 17:16:12 GMT
So, this basically means that Flume's transactional model is also
unreliable. That would have to mean that the downstream agent is sending an
ack to the upstream agent before it actually persists the event.

On Apr 7, 2014 10:48 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
>
> No. If you need to guarantee delivery of events please use a file channel.
> https://blogs.apache.org/flume/entry/apache_flume_filechannel
>
>
> On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon <cshannon108@gmail.com>
wrote:
>>
>>
>> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
>> >
>> >
>> >
>> >
>> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <
durgapalmohit@gmail.com> wrote:
>> >>
>> >> Hi Jeff,
>> >>
>> >> Yes, I am using the memory channel, and that's because I want it to
be more reliable and not miss any events/messages.
>> >> As I've read in flume documentation that the memory channel is fast
but there could be a chance of missing events if the in-memory buffer fills
up.
>> >
>> >
>> > Memory channel is not reliable, meaning if the flume agent goes down
or is restarted while there are events in the channel than this data will
be lost.
>> > For reliability please use the file channel.
>> >
>>
>> Jeff,
>>
>> I am using an upstream agent with a spooling directory source and a
memory channel, and the downstream agent uses a memory channel and an HDFS
sink. If my downstream agent goes down for any reason, are the entries lost
in the downstream agent's memory channel still preserved in the memory
channel / file directory of the upstream agent?
>>
>> All the best, Chris
>
>
 No. If you need to guarantee delivery of events please use a file channel.
https://blogs.apache.org/flume/entry/apache_flume_filechannel


On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon
<cshannon108@gmail.com>wrote:

>
> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
> >
> >
> >
> >
> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <durgapalmohit@gmail.com>
> wrote:
> >>
> >> Hi Jeff,
> >>
> >> Yes, I am using the memory channel, and that's because I want it to be
> more reliable and not miss any events/messages.
> >> As I've read in flume documentation that the memory channel is fast but
> there could be a chance of missing events if the in-memory buffer fills up.
> >
> >
> > Memory channel is not reliable, meaning if the flume agent goes down or
> is restarted while there are events in the channel than this data will be
> lost.
> > For reliability please use the file channel.
> >
>
> Jeff,
>
> I am using an upstream agent with a spooling directory source and a memory
> channel, and the downstream agent uses a memory channel and an HDFS sink.
> If my downstream agent goes down for any reason, are the entries lost in
> the downstream agent's memory channel still preserved in the memory channel
> / file directory of the upstream agent?
>
> All the best, Chris
>

Mime
View raw message