Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Received-SPF: pass (athena.apache.org: domain of jay.kreps@gmail.com
 designates 209.85.210.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOeJiJiiw0=XeEqxgC31-f0Zy93XR=Z8Ah4SqN8kq4UoVHiRrQ@mail.gmail.com>
References: 
 <CAOeJiJiiw0=XeEqxgC31-f0Zy93XR=Z8Ah4SqN8kq4UoVHiRrQ@mail.gmail.com>
Date: Mon, 11 Feb 2013 12:08:45 -0800
Message-ID: 
 <CAOeJiJj=X9giGOJSW1GFTdgZzAPeunqht880O7_ewKKRedPUOg@mail.gmail.com>
Subject: Re: admin functionality refactoring
From: Jay Kreps <jay.kreps@gmail.com>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Content-Type: multipart/alternative; boundary=e89a8f2357a1efd1d904d578792e

--e89a8f2357a1efd1d904d578792e
Content-Type: text/plain; charset=ISO-8859-1

One thought I had on some of these things was that it is actually kind of
funny that we implement them in zookeeper. Technically we could implement
them in Kafka itself which would mean less code (potentially) and also a
clean api to clients.

The idea is that many of the admin apis are basically changing cluster
state. Examples are adding a topic, deleting a topic, altering the log
config for a topic, changing the replication factor for a topic, migrating
partitions, etc.

One way to implement these would be to define an admin topic in kafka. Then
standardize a JSON format or something for these command type messages. The
API to make these changes would just be to publish a message to this topic.
This would allow all brokers to subscribe and carry out these commands in
order, and would allow any client that has support for the producer api to
publish commands.

I think in addition to providing an API this is actually the functionality
needed for most of these cluster changes. For example the dynamic config
patch I posted is basically implementing a queue like this but in zk for
subscribing to and applying config changes.

This is not fully thought through, but thought I would just toss the idea
out there.

-Jay


On Fri, Feb 8, 2013 at 3:12 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> Admin functionality in 0.8 is kind of messy.  What should we do?
>
> There are a bunch of misc. command line tools under kafka.admin.
> AdminUtils is the closest thing we have to an admin interface, but it is
> weirdly implemented as all static methods. Furthermore its APIs are really
> wonky and tend to describe the operation they perform in zookeeper rather
> than the high-level thing they accomplish.
>
> Since people are going to want programmatic access to administrative
> functionality it would be good to think this through a bit. The least we
> could do would be to refactor AdminUtils into an Admin class and think
> through those APIs enough to make them usable from java/scala. This would
> mean having a clear API for each thing that the various tools currently do
> (create/delete/alter/describe/list topics, migrate partitions, shutdown
> broker, etc).
>
> Arguably some of these should really be RPC apis so other languages can
> invoke basic operations like creating and deleting topics. But this is not
> critical so we can probably do it later.
>
> I think it would be good to also think about how we ended up here. I think
> we are not putting in place basic APIs that provide a layer of
> simplification for common operations. Instead we tend to just be jabbing
> directly at Zookeeper from different places using ad hoc methods in
> ZkUtils. This is really hard to understand or change. In general we seem to
> have trouble thinking through APIs.
>
> Thoughts?
>
> -Jay
>
>

--e89a8f2357a1efd1d904d578792e--