Mailing-List: contact users-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@kafka.apache.org
Received-SPF: pass (nike.apache.org: domain of junrao@gmail.com designates
 209.85.219.52 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <52B290690412D549866EE6AD16716DFE29B60161F5@Olympus.visibletech.net>
References: 
 <52B290690412D549866EE6AD16716DFE29B60161F5@Olympus.visibletech.net>
Date: Mon, 11 Feb 2013 21:39:19 -0800
Message-ID: 
 <CAFbh0Q3fyM1zY46sAyE3Ygd-oXAi2D1mHYQ9DgN5w6mV47pj-A@mail.gmail.com>
Subject: Re: Producers errors when failing a broker in a replicated 0.8
 cluster.
From: Jun Rao <junrao@gmail.com>
To: users@kafka.apache.org
Content-Type: multipart/alternative; boundary=e89a8ff1c2e876976804d58072cc

--e89a8ff1c2e876976804d58072cc
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Bob,

In 0.8, if you send a set of messages in sync mode, the producer will throw
back an exception if at least one message can't be sent to the broker after
all retries. The client won't know which messages are sent successfully and
which are not. We do plan to improve the producer API after 0.8 that can
expose more information to the client.

Thanks,

Jun

On Mon, Feb 11, 2013 at 9:27 AM, Bob Jervis <bjervis@visibletechnologies.co=
m
> wrote:

> We are in final testing of Kafka and so far the fail-over tests have been
> pretty encouraging.  If we kill (-9) one of two kafka brokers, with
> replication factor=3D2 we see a flurry of activity as the producer fails =
and
> retries its writes (we use a bulk, synchronous send of 1000 messages at a
> time, each message ~1K long).  Sometimes the library finds the newly
> elected leader before returning to the application and sometimes it
> doesn=92t.  We added retry/backoff logic to our code and we don=92t seem =
to be
> losing content.****
>
> ** **
>
> However, we have another app in the pipeline that does a fan-out from one
> Kafka topic to dozens of topics.  We still use a single, synchronous, bul=
k
> send.****
>
> ** **
>
> My question is what are the semantics of a bulk send like that, where one
> broker dies, but the topic leaders have been spread across both brokers.
> Do we get any feedback on which messages went through and which were
> dropped because the leader just died?  For our own transactioning we can
> mark messages as =91retries=92 if we suspect there might have been any
> hanky-panky, but if we can reliably avoid extra work by not re-sending
> messages that we know have been delivered we can avoid the extra work on
> the client side.****
>
> ** **
>
> Thanks for any insight,****
>
> ** **
>
> *Bob Jervis | Senior Architect*
>
>
> *[image: Description: Description: Visible-sm]*<http://www.visibletechnol=
ogies.com/>
> **
>
> Seattle *| *Boston* | *New York *|* London****
>
> *Phone:* 425.957.6075* | Fax:* 781.404.5711 ****
>
> ** **
>
> *Follow Visibly Intelligent Blog<http://www.visibletechnologies.com/blog/=
>
> *
>
> ** **
>
> [image: Description: Description: LinkedIn_Logo60px[1]]<http://twitter.co=
m/visible>[image:
> Description: Description: facebook]<http://www.facebook.com/Visible.Techn=
ologies>
>  [image: Description: Description: in]<http://www.linkedin.com/company/vi=
sible-technologies>
> ****
>
> ** **
>

--e89a8ff1c2e876976804d58072cc--