flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Flume Configuration & topology approach
Date Mon, 07 Apr 2014 17:38:55 GMT
No not at all. Flume's transactional model guarantees delivery between hops.
https://blogs.apache.org/flume/entry/flume_ng_architecture


On Mon, Apr 7, 2014 at 10:16 AM, Christopher Shannon
<cshannon108@gmail.com>wrote:

> So, this basically means that Flume's transactional model is also
> unreliable. That would have to mean that the downstream agent is sending an
> ack to the upstream agent before it actually persists the event.
>
> On Apr 7, 2014 10:48 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
> >
> > No. If you need to guarantee delivery of events please use a file
> channel.
> > https://blogs.apache.org/flume/entry/apache_flume_filechannel
> >
> >
> > On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon <
> cshannon108@gmail.com> wrote:
> >>
> >>
> >> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <
> durgapalmohit@gmail.com> wrote:
> >> >>
> >> >> Hi Jeff,
> >> >>
> >> >> Yes, I am using the memory channel, and that's because I want it to
> be more reliable and not miss any events/messages.
> >> >> As I've read in flume documentation that the memory channel is fast
> but there could be a chance of missing events if the in-memory buffer fills
> up.
> >> >
> >> >
> >> > Memory channel is not reliable, meaning if the flume agent goes down
> or is restarted while there are events in the channel than this data will
> be lost.
> >> > For reliability please use the file channel.
> >> >
> >>
> >> Jeff,
> >>
> >> I am using an upstream agent with a spooling directory source and a
> memory channel, and the downstream agent uses a memory channel and an HDFS
> sink. If my downstream agent goes down for any reason, are the entries lost
> in the downstream agent's memory channel still preserved in the memory
> channel / file directory of the upstream agent?
> >>
> >> All the best, Chris
> >
> >
>  No. If you need to guarantee delivery of events please use a file
> channel.
> https://blogs.apache.org/flume/entry/apache_flume_filechannel
>
>
> On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon <cshannon108@gmail.com
> > wrote:
>
>>
>> On Apr 7, 2014 9:35 AM, "Jeff Lord" <jlord@cloudera.com> wrote:
>> >
>> >
>> >
>> >
>> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal <
>> durgapalmohit@gmail.com> wrote:
>> >>
>> >> Hi Jeff,
>> >>
>> >> Yes, I am using the memory channel, and that's because I want it to be
>> more reliable and not miss any events/messages.
>> >> As I've read in flume documentation that the memory channel is fast
>> but there could be a chance of missing events if the in-memory buffer fills
>> up.
>> >
>> >
>> > Memory channel is not reliable, meaning if the flume agent goes down or
>> is restarted while there are events in the channel than this data will be
>> lost.
>> > For reliability please use the file channel.
>> >
>>
>> Jeff,
>>
>> I am using an upstream agent with a spooling directory source and a
>> memory channel, and the downstream agent uses a memory channel and an HDFS
>> sink. If my downstream agent goes down for any reason, are the entries lost
>> in the downstream agent's memory channel still preserved in the memory
>> channel / file directory of the upstream agent?
>>
>> All the best, Chris
>>
>
>

Mime
View raw message