fluo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: [DISCUSS] Separate deployment code from Fluo distribution tarball
Date Tue, 12 Jul 2016 18:40:21 GMT
Minimally we should create an issue and link to this thread.   One thing we
could do is create a blog post about the things we would like to do for
1.1.0 after the 1.0.0 release.

On Tue, Jul 12, 2016 at 2:34 PM, Mike Walch <mwalch@apache.org> wrote:

> I will definitely update our documentation when this work is completed.  We
> do have an architecture page (
> https://fluo.apache.org/docs/fluo/1.0.0-beta-2/architecture/) but it will
> need to be updated and could be expanded.  Are you also looking for this
> design to be documented before work is started on it?
>
> On Tue, Jul 12, 2016 at 2:06 PM Josh Elser <josh.elser@gmail.com> wrote:
>
> > +1 to that succinctness
> >
> > Any chance I could persuade you into writing some of the plan down
> > somewhere (maybe on the website)? It would be nice to have a
> > clean/defined architecture for Fluo. Would help attract new contributors
> > (easier to learn about how Fluo works and its modules).
> >
> > Christopher wrote:
> > > I like the idea of minimizing the core to increase packaging/deployment
> > > options, as well as the idea of bootstrapping a few possible deployment
> > > strategies.
> > >
> > > On Tue, Jul 12, 2016 at 1:04 PM Keith Turner<keith@deenlo.com>  wrote:
> > >
> > >> I am in favour of creating multiple separate projects for launch Fluo
> > >> workers and oracle in different environments.   We should do this in
> > such a
> > >> way that these projects only use Fluo Core public APIs.
> > >>
> > >> For the 1.0.0 we can proceed w/ the baked in Twill+YARN support.  For
> > 1.1.0
> > >> we could work towards externalizing this launch functionality and
> adding
> > >> any new public APIs needed.  The 1.1.0 release could deprecate the
> > built in
> > >> YARN+Twill functionality.
> > >>
> > >> On Tue, Jul 12, 2016 at 11:59 AM, Mike Walch<mwalch@apache.org>
> wrote:
> > >>
> > >>> The Fluo distribution tarball currently contains code that allows
> users
> > >> to
> > >>> start Fluo applications in YARN using Twill.  While deploying to YARN
> > has
> > >>> worked well, Fluo should not be tied to a single cluster resource
> > >> manager.
> > >>> For example, users could have problems deploying their Fluo
> application
> > >> to
> > >>> YARN with our current setup as Twill brings in dependencies that can
> > >>> conflict with the user-provided dependencies in their Fluo
> application.
> > >>>
> > >>> In order to give users deployment options, we should remove any
> > >> deployment
> > >>> code from our tarball.  We should start releasing a simpler tarball
> > that
> > >>> contains only the necessary jars and commands to configure a Fluo
> > >>> application (in Zookeeper) and start the processes of a Fluo
> > application
> > >>> locally.  It looks Kafka Streams is already taking this approach.
> Read
> > >> the
> > >>> section 'Docker, Mesos, and Kubernetes, Oh My!' in the blog post
> below:
> > >>>
> > >>>
> > >>>
> > >>
> >
> http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple
> > >>> If we also took this the approach, the commands and structure of the
> > >>> tarball would need to follow SEMVER as users and downstream projects
> > >> would
> > >>> depend on them.  Users could then either script deployments using
> Chef,
> > >>> Ansible, Salt, etc. or use downstream projects created to make it
> easy
> > to
> > >>> deploy Fluo applications to YARN, Mesos, Kubernetes, etc.  We could
> > >>> initially create and maintain some these downstream projects.
> However,
> > >>> they could innovate/survive based on community interset.
> > >>>
> > >>> After we release 1.0.0, I would like to start working in this
> direction
> > >> for
> > >>> Fluo. I am interested to see if anyone has any views to share on this
> > >>> topic.
> > >>>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message