flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bhaskar V. Karambelkar" <bhaska...@gmail.com>
Subject Re: Proper documentation for setting up sink groups
Date Thu, 23 Aug 2012 18:43:10 GMT
Hi Hari,
Yes I did read the whole guide end to end.
But I still have doubts

The fact that multiple sinks can feed from the same channel is news to me.
I don't see it explicitly mentioned in the docs,
so i guess I assumed wrongly, that only one sink can feed from a channel.

a)Can you explain in detail , how having multiple sinks taking events from
one channel, is useful in a "fast source slow sinks" scenario ?

b) Also if I read your explanation below correctly there are 3 possible

1) multiple sinks feeding from a single channel , with the default sink
processor this will be like a multiplexing channel with all sinks getting
all the events that come in the channel.

2) multiple sinks feeding from a single channel , with fail_over sink
processor, only one sink will get the events at a give time, with flume
failing over to next available sink in case the first one fails ?

3) multiple sinks feeding from a single channel, with
load balancing processor, with all sinks getting events in a
round-robin/random order.

Is this a correct assumption ? I am aware of #2 and #3, not sure about #1.

On Thu, Aug 23, 2012 at 12:43 PM, Hari Shreedharan <
hshreedharan@cloudera.com> wrote:

> Did you read this:
> http://flume.apache.org/FlumeUserGuide.html#flume-sink-processors
> That explains how to use sink groups. Also there is nothing wrong with
> multiple sinks taking events from one channel. This is an especially useful
> configuration if you have a very fast source and much slower sinks.
> Hari
> --
> Hari Shreedharan
> On Thursday, August 23, 2012 at 9:28 AM, Bhaskar V. Karambelkar wrote:
> > The sink group document doesn't mention anything about how
> > to hook up sink groups to the rest of the config in order to work.
> >
> > e.g. under normal circumstances one channel is linked with one sink.
> >
> > But for failover sink group , looks like both the sinks should be hooked
> up to the same channel,
> > but this is not mentioned any where.
> >
> > Similarly, what exactly needs to be done for load balancing sink ?
> >
> > thanks

View raw message