Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
MIME-Version: 1.0
In-Reply-To: <54F40977-4B8B-4033-BD37-6E88F371CD8D@gmail.com>
References: <5C07F644-F9B2-4584-8663-82C768B1BE1B@gmail.com>
	<CADQEQnhB8B2m1aRkJ=cbEx1tBXYXD=vLGwy375fQMx8vX2iDMg@mail.gmail.com>
	<CANkRryDoNWEGb8cCzMHVe89ir-qMESY7=SeajGz97mESzaxkGg@mail.gmail.com>
	<CALmgQN75iEM62qxC57v19b_3KCBaGdzKST28K7wnkizbO1ssiA@mail.gmail.com>
	<CANkRryBQhomM9N3S42ibHV2yTqaGG3FDaFTgdYhNuWZ+LnaChQ@mail.gmail.com>
	<CALmgQN777SEdqWNYQL28SXah-AKrJj-qnYn6ngzWDNnbqSymYg@mail.gmail.com>
	<54F40977-4B8B-4033-BD37-6E88F371CD8D@gmail.com>
Date: Sun, 5 Jul 2015 22:14:36 -0700
Message-ID: 
 <CAHBV8WcY1qRLqmZQwCB+v=UjdpCtxWHWGcDNC7QWr0wKVZtBmg@mail.gmail.com>
Subject: Re: Flafka: Kafka channel unable to deliver event
From: Gwen Shapira <gshapira@cloudera.com>
To: user@flume.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

See inline :)

On Sun, Jul 5, 2015 at 5:37 PM, Jun MA <mj.saber1990@gmail.com> wrote:
> Hi Rufus,
>
> Thank you so much for your help, I do bypass the issue.
>
> Another question I have is that can I have two Flafka agents consume from
> one topic (agents run on same machine or different machine)?

Yes.

> Will the two
> agents have exact same two copies or each agent will consume one part of =
the
> topic?

It depends on how you configure them.
If you configure two different consumer group IDs, they will each get a cop=
y.
If they use the same consumer group ID, each agent will consume from
different subset of partitions. They will automatically load balance
partitions in case more agents join or if agents are stopped.

> The purpose I want to run two agents on same topic is that I want to
> have a high availability. What I want is that two agents will consume
> different portion of the topic but if one agent down, the other one will
> consume everything from the topic.
> From what I tested, I bring up two agents on same machine with different
> agent name (everything else are the same), and only the first one can
> consume messages from topic. I=E2=80=99m wondering if it is the right beh=
avior. If
> so, is there anyway I can solve this single point of failure?

If you configure both Flafkas with same consumer group ID, you should
get the behavior you want.
Just make sure you have at least two partitions - the load balancing
behavior happens at the partition level, so you can only have as many
concurrent agents as you have partitions.

Gwen

>
> Bests,
> Jun
>
> On Jul 4, 2015, at 10:22 AM, Johny Rufus <jrufus@cloudera.com> wrote:
>
> Been looking into this for some time and found a couple of issues. Raised
> Jiras for both, there are workarounds to get past these errors.
>
> https://issues.apache.org/jira/browse/FLUME-2734
> https://issues.apache.org/jira/browse/FLUME-2735
>
> For timeout to take effect, you need to specify as -
> agent1.channels.channel1.timeout =3D 1000000 (This is a temporary work ar=
ound,
> kafka.consumer.timeout.ms should work as per the guide when FLUME-2734 is
> resolved)
>
> To get past the IllegalStateException, I had to download the zookeeper ja=
r
> from http://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper/3.3=
.6
> and put it in the lib directory. This is also a workaround to get past th=
e
> issue, until we figure out the root cause.
>
>
> Thanks,
> Rufus
>
> On Fri, Jul 3, 2015 at 10:09 PM, Jun Ma <mj.saber1990@gmail.com> wrote:
>>
>> Oops, thanks for figuring that out.
>> Any idea why unable to deliver event?
>>
>> On Fri, Jul 3, 2015 at 8:00 PM, Johny Rufus <jrufus@cloudera.com> wrote:
>>>
>>> There is a typo in your property "cnannel1", hence the property is not
>>> set
>>>
>>> a1.channels.cnannel1.kafka.consumer.timeout.ms =3D 1000000
>>>
>>> Thanks,
>>> Rufus
>>>
>>> On Fri, Jul 3, 2015 at 4:49 PM, Jun Ma <mj.saber1990@gmail.com> wrote:
>>>>
>>>> Thanks for your reply. But from what I read, the magic things Flafka
>>>> does is that you don't need to have a source, you can directly move th=
ings
>>>> from channel to sink.
>>>>
>>>> http://blog.cloudera.com/blog/2014/11/flafka-apache-flume-meets-apache=
-kafka-for-event-processing/
>>>> (see Flume's kafka channel)
>>>>
>>>>
>>>> On Fri, Jul 3, 2015 at 1:26 PM, Foo Lim <foo.lim@vungle.com> wrote:
>>>>>
>>>>> There's an error that you didn't specify a source:
>>>>>
>>>>> Agent configuration for 'a1' has no sources.
>>>>
>>>>
>>>
>>
>
>