flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mongeol Heo <mongeol....@gmail.com>
Subject Re: Kafka version
Date Mon, 27 Oct 2014 08:34:56 GMT
Hello, Gwen.

I have tried flafka, and used it as a sink for producing logs to kafka
cluster.
It produced 1M logs in few seconds which is pretty fast.
However, I have a problem here.
It produced all logs into one partition, even if there are three partitions.
What I want is that 1M logs are well distributed through the three
partitions, so I can have the benefit of using three consumers to improve
the consume speed.
Is there any way to get what I want?
I am going to try 'topic.metadata.refresh.interval.ms' option, but I
believe it will not work as I expect.

Thanks.

On Sat, Oct 25, 2014 at 2:50 AM, Gwen Shapira <gshapira@cloudera.com> wrote:

> Its not in the user guide because the user guide refers to releases
> and Flafka is only on the unreleased trunk right now.
>
> On Fri, Oct 24, 2014 at 1:08 AM, Mongeol Heo <mongeol.heo@gmail.com>
> wrote:
> > I wonder why flume 1.5.0.1 user guide, which shows below, does not
> include
> > it although the link you give is for 1.5.0, and I think this is the one
> > which is mentioned by Gwen Shapira above.
> > Thank you.
> >
> > http://flume.apache.org/FlumeUserGuide.html
> >
> > On Fri, Oct 24, 2014 at 4:55 PM, Hari Shreedharan
> > <hshreedharan@cloudera.com> wrote:
> >>
> >>
> >>
> http://archive.cloudera.com/cdh5/cdh/5/flume-ng/FlumeUserGuide.html#kafka-source
> >>
> >> This will come in the next Apache release.
> >>
> >>
> >> On Fri, Oct 24, 2014 at 12:28 AM, Mongeol Heo <mongeol.heo@gmail.com>
> >> wrote:
> >>>
> >>> Hi, Hari.
> >>> Could you point out one of the kafka source which supports version
> 0.8+?
> >>> since all I found are for 0.7+.
> >>> Thanks,
> >>>
> >>> Mungeol
> >>>
> >>> On Fri, Oct 24, 2014 at 4:15 PM, Hari Shreedharan
> >>> <hshreedharan@cloudera.com> wrote:
> >>>>
> >>>> There is a Kafka Source too, but both are not yet released in a Flume
> >>>> release, though one will hopefully be.
> >>>>
> >>>> Thanks,
> >>>> Hari
> >>>>
> >>>>
> >>>> On Thu, Oct 23, 2014 at 11:53 PM, Mangtani, Kushal
> >>>> <Kushal.Mangtani@viasat.com> wrote:
> >>>>>
> >>>>> Yes, Flume has support for Kafka 0.8.1. Refer the below link for
> Flume
> >>>>> Kafka sink,source. I have been using this in my prod and it is
> relatively
> >>>>> stable
> >>>>> https://github.com/thilinamb/flume-ng-kafka-sink
> >>>>>
> >>>>> -Kushal Mangtani
> >>>>> ________________________________
> >>>>> From: Mongeol Heo [mongeol.heo@gmail.com]
> >>>>> Sent: Thursday, October 23, 2014 11:38 PM
> >>>>> To: user@flume.apache.org
> >>>>> Subject: Kafka version
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> My question is that will flume sink and source for apache kafka
> support
> >>>>> latest version of it?
> >>>>> As I know, even sink/source for kafka 0.7.2 is not released yet.
> >>>>> It will be really great if it does.
> >>>>> Thanks.
> >>>>>
> >>>>> Best regards,
> >>>>>
> >>>>> - Mungeol
> >>>>
> >>>>
> >>>
> >>
> >
>

Mime
View raw message