Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Received-SPF: pass (athena.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: 
 <CAFbh0Q0RS36XF_89X=o+GeWDLHz2vBmtKXmtEJX8d3mt=upaWQ@mail.gmail.com>
References: 
 <CAA7ooCDtskcyPVL8uCY6=FFTtsKLTKMhGnABHS+DmKENFba3_Q@mail.gmail.com>
	<CAFbh0Q3tdTgZmq4ZK6ycJgiyc_A4XJBEDVyERd+Xs0nvaqpsVQ@mail.gmail.com>
	<CAA7ooCCXqUO_oWQc03Ymjc9BHZSERSduH+wZ_e_tymgdj3bbLg@mail.gmail.com>
	<CAA7ooCB68LucNL+_LU3jYSD5kUpQT+y0vRTMm12zOjA0isYruQ@mail.gmail.com>
	<CAFbh0Q0RS36XF_89X=o+GeWDLHz2vBmtKXmtEJX8d3mt=upaWQ@mail.gmail.com>
Date: Wed, 22 Jan 2014 17:36:16 -0500
Message-ID: 
 <CAA7ooCBUNTa-M=uk64gAsT1DqGi4-cONLyTZ6vmYexMzi_pmEQ@mail.gmail.com>
Subject: Re: Kafka server start from resource negotiator(s)
From: Joe Stein <joe.stein@stealth.ly>
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Content-Type: multipart/alternative; boundary=089e01229a06ca024804f096c0af

--089e01229a06ca024804f096c0af
Content-Type: text/plain; charset=ISO-8859-1

Hi Jun, see inline

On Tue, Jan 21, 2014 at 1:14 PM, Jun Rao <junrao@gmail.com> wrote:

> Hi, Joe,
>
> Thanks for the update. This looks great. A couple of questions and
> comments.
>
> 1. How general is the framework? For example, if I want to add the support
> for Apache Yarn, is anything there can be reused?
>

I don't think there is going to be much to share but I am honestly not far
enough along to know for sure.  I opened KAFKA-1206 for any (if any)
changes that they would both need or use to make it work from the existing
code (so we can launch brokers, producers and consumers in an abstract way
not just from shell scripts but from the resource negotiator system.  Quite
possible no change required...


>
> 2. My current feeling is that it's probably better to maintain those
> container supports outside Kafka's git repo (e.g., can be on github).
> People may be interested in integrating Kafka with Mesos, Yarn, VMs on EC2,
> Azure, and Rackspace, etc. Maintaining all those in contrib may not be a
> scalable solution.
>

I am neutral either way on this at the moment.  Right now, I am just
thinking about getting it working :)
If it lives only @ https://github.com/stealthly/kafka-mesos that is not a
bad thing (like how we do client libraries).


>
> Thanks,
>
> Jun
>
>
> On Sat, Jan 18, 2014 at 5:48 AM, Joe Stein <joe.stein@stealth.ly> wrote:
>
> > I got through some good research and testing this week on getting Kafka
> to
> > run on Mesos.
> >
> > I updated the Mesos ticket
> > https://issues.apache.org/jira/browse/KAFKA-1207with some thoughts.
> >
> > Next step I am going to draw out some diagrams and update the wiki page.
>  I
> > was thinking of writing this in the contrib folder and adding two bin
> > scripts if folks are interested in having this as part of the overall
> > project.  There are few minor changes (like what I put in 1206) that
> could
> > occur but for now I can get it working on 0.8.0 I believe (refactoring
> > afterwards some changes non resource negotiator specific).
> >
> > For now I am going to work on it here
> > https://github.com/stealthly/kafka-mesos (nothing there just yet) and
> once
> > parts of it are working correctly upload patches to JIRA.
> >
> > Thoughts? Suggestions? Feedback? Tomatoes?
> >
> >
> > /*******************************************
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> >
> > On Thu, Jan 16, 2014 at 12:43 AM, Joe Stein <joe.stein@stealth.ly>
> wrote:
> >
> > > Right now I am testing it that way. Once it is working and if that
> > becomes
> > > a problem then moving it into a mesosFramework project is no problem.
> > if
> > > we do that it would be beneficial if they were organized under some
> > > resource manager folder or something.
> > >
> > > The way I wrote https://issues.apache.org/jira/browse/KAFKA-1206 was
> > with
> > > that in mind so folks have a light interface for doing this.  The
> > specific
> > > systems resource negotiator implementation could go in their project
> and
> > > just importing the ResourceNegotiator trait in the project's reference
> of
> > > the jar.
> > >
> > > /*******************************************
> > >  Joe Stein
> > >  Founder, Principal Consultant
> > >  Big Data Open Source Security LLC
> > >  http://www.stealth.ly
> > >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > ********************************************/
> > >
> > >
> > > On Thu, Jan 16, 2014 at 12:23 AM, Jun Rao <junrao@gmail.com> wrote:
> > >
> > >> Hi, Joe,
> > >>
> > >> Thanks for looking into this. Do you plan to add a dependency to Mesos
> > in
> > >> kafka jar?
> > >>
> > >> Jun
> > >>
> > >>
> > >> On Wed, Jan 15, 2014 at 10:50 AM, Joe Stein <joe.stein@stealth.ly>
> > wrote:
> > >>
> > >> > Hi, I am working at adding some code to the kafka.Kafka.scala file.
> > >> >
> > >> > What I was thinking for the Utils.loadProps is to create some
> > >> > ResourceNegotiatedConfiguration class or something to sit with it so
> > >> > various frameworks can be supported.
> > >> >
> > >> > For Mesos I am thinking of storing the properties in zookeeper. For
> a
> > >> new
> > >> > node joining => registering a default config (originally set by a
> > tool)
> > >> and
> > >> > saving it unique for that broker in a znode.  This znode would be
> the
> > >> input
> > >> > to the serverConfig for the KafkaServerStartable.
> > >> >
> > >> > I am going to create a kafka.tools.ResourceNegotiator.ApacheMesos
> tool
> > >> too.
> > >> >  that tool will take in a server.properties file to set the original
> > >> > default. Once loaded in zookeeper another command to take a property
> > and
> > >> > flag it with a function to run (like the "use new broker id value")
> > and
> > >> > such.
> > >> >
> > >> > I would rather have some implementation in Kafka.scala and have
> object
> > >> > KafkaMesos be in Kafka.scala too but wasn't sure what other thoughts
> > >> folks
> > >> > might have?
> > >> >
> > >> > I will be creating JIRA soon for it.
> > >> >
> > >> > /*******************************************
> > >> >  Joe Stein
> > >> >  Founder, Principal Consultant
> > >> >  Big Data Open Source Security LLC
> > >> >  http://www.stealth.ly
> > >> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > >> > ********************************************/
> > >> >
> > >>
> > >
> > >
> >
>

--089e01229a06ca024804f096c0af--