Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Date: Thu, 5 Jan 2012 19:06:39 +0000 (UTC)
From: "Jun Rao (Commented) (JIRA)" <jira@apache.org>
To: kafka-dev@incubator.apache.org
Message-ID: 
 <943426546.11051.1325790399496.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (KAFKA-50) kafka intra-cluster replication
 support
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/KAFKA-50?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D131807=
12#comment-13180712 ]=20

Jun Rao commented on KAFKA-50:
------------------------------

Here is a breakdown of all the jiras and their dependencies:

1. kafka-47: create/delete data structures in ZK, automatically create topi=
c, and use them (making partitions logical, supporting only 1 replica, no f=
ailure support). =EF=BB=BF=EF=BB=BF(L1)
  1.1 kafka-237: create/delete ZK path for a topic in an admin tool (L0)
  1.2 kafka-238: add a getTopicMetaData method in broker and expose it to p=
roducer
  1.3 kafka-239: Wire existing producer and consumer to use the new ZK data=
 structure
2. kafka-202: decouple request handler and socket sever; enabling long poll=
 in the consumer (L1)
3. kafka-240: implement new producer and consumer request format (L1)=20
4. kafka-49: add ack to ProduceRequest (L2). depending on #3
5. kafka-48: long poll in consumer (L2). depending on #2
6. kafka-44: commit thread, replica fetcher thread (L3). depending on #1, #=
4, #5
7. kafka-45: broker starting up, leader election (L3). depending on #1
8. kafka-46: various ZK listeners (L3). depending on #1
9. kafka-43: move master to preferred replica when possible (L4). optimizat=
ion
10. kafka-42: rebalance partition with new brokers (L4). extra feature

               =20
> kafka intra-cluster replication support
> ---------------------------------------
>
>                 Key: KAFKA-50
>                 URL: https://issues.apache.org/jira/browse/KAFKA-50
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Jun Rao
>            Assignee: Jun Rao
>             Fix For: 0.8
>
>         Attachments: kafka_replication_detailed_design_v2.pdf, kafka_repl=
ication_highlevel_design.pdf, kafka_replication_lowlevel_design.pdf
>
>
> Currently, Kafka doesn't have replication. Each log segment is stored in =
a single broker. This limits both the availability and the durability of Ka=
fka. If a broker goes down, all log segments stored on that broker become u=
navailable to consumers. If a broker dies permanently (e.g., disk failure),=
 all unconsumed data on that node is lost forever. Our goal is to replicate=
 every log segment to multiple broker nodes to improve both the availabilit=
y and the durability.=20
> We'd like to support the following in Kafka replication:=20
> 1. Configurable synchronous and asynchronous replication=20
> 2. Small unavailable window (e.g., less than 5 seconds) during broker fai=
lures=20
> 3. Auto recovery when a failed broker rejoins=20
> 4. Balanced load when a broker fails (i.e., the load on the failed broker=
 is evenly spread among multiple surviving brokers)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira