Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Date: Fri, 20 Jan 2012 22:36:41 +0000 (UTC)
From: "Joel Koshy (Created) (JIRA)" <jira@apache.org>
To: kafka-dev@incubator.apache.org
Message-ID: 
 <173678975.62027.1327099001326.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Created] (KAFKA-249) Separate out Kafka mirroring into a
 stand-alone app
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Separate out Kafka mirroring into a stand-alone app
---------------------------------------------------

                 Key: KAFKA-249
                 URL: https://issues.apache.org/jira/browse/KAFKA-249
             Project: Kafka
          Issue Type: Improvement
          Components: core
            Reporter: Joel Koshy
            Assignee: Joel Koshy
             Fix For: 0.7.1


I would like to discuss on this jira, the feasibility/benefits of separating
out Kafka's mirroring feature from the broker into a stand-alone app, as it
currently has a couple of limitations and issues.

For example, we recently had to deal with Kafka mirrors that were in fact
idle due to the fact that mirror threads were not created at start-up due to
a rebalancing exception, but the Kafka broker itself did not shutdown. This
has since been fixed, but is indicative of (avoidable) problems in embedding
non-broker specific features in the broker.

Logically, it seems to make sense to separate it out to achieve better
division of labor.  Furthermore, enhancements to mirroring may be less
clunky to implement and use with a stand-alone app.  For example to support
custom partitioning on the target cluster, or to mirror from multiple
clusters we would probably need to be able to pass in multiple embedded
consumer/embedded producer configs, which would be less ugly if the
mirroring process were a stand-alone app.  Also, if we break it out, it
would be convenient to use as a "consumption engine" for the console
consumer which will make it easier to add on features such as wildcards in
topic consumption, since it contains a ZooKeeper topic discovery component.

Any suggestions and/or objections to this?


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira