bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From RJ Nowling <rnowl...@gmail.com>
Subject Re: What will the next generation of bigtop look like?
Date Mon, 08 Dec 2014 00:04:35 GMT
Which other projects depend on BigTop?  How will the questions about the
direction of BigTop affect those projects?

On Sun, Dec 7, 2014 at 6:10 PM, Roman Shaposhnik <roman@shaposhnik.org>
wrote:

> Hi!
>
> On Sat, Dec 6, 2014 at 3:23 PM, jay vyas <jayunit100.apache@gmail.com>
> wrote:
> > hi bigtop !
> >
> > I thought id start a thread a few vaguely related thoughts i have around
> > next couple iterations of bigtop.
>
> I think in general I see two major ways for something like
> Bigtop to evolve:
>    #1 remain a 'box of LEGO bricks' with very little opinion on
>         how these pieces need to be integrated
>    #2 start driving oppinioned use-cases for the particular kind of
>         bigdata workloads
>
> #1 is sort of what all of the Linux distros have been doing for
> the majority of time they existed. #2 is close to what CentOS
> is doing with SIGs.
>
> Honestly, given the size of our community so far and a total
> lack of corporate backing (with a small exception of Cloudera
> still paying for our EC2 time) I think #1 is all we can do. I'd
> love to be wrong, though.
>
> > 1) Hive:  How will bigtop to evolve to support it, now that it is much
> more
> > than a mapreduce query wrapper?
>
> I think Hive will remain a big part of Hadoop workloads for forseeable
> future. What I'd love to see more of is rationalizing things like how
> HCatalog, etc. need to be deployed.
>
> > 2) I wonder wether we should confirm cassandra interoperability of spark
> in
> > bigtop distros,
>
> Only if there's a significant interest from cassandra community and even
> then my biggest fear is that with cassandra we're totally changing the
> requirements for the underlying storage subsystem (nothing wrong with
> that, its just that in Hadoop ecosystem everything assumes very HDFS'ish
> requirements for the scale-out storage).
>
> > 4) in general, i think bigtop can move in one of 3 directions.
> >
> >   EXPAND ? : Expanding to include new components, with just basic
> interop,
> > and let folks evolve their own stacks on top of bigtop on their own.
> >
> >   CONTRACT+FOCUS ?  Contracting to focus on a lean set of core
> components,
> > with super high quality.
> >
> >   STAY THE COURSE ? Staying the same ~ a packaging platform for just
> > hadoop's direct ecosystem.
> >
> > I am intrigued by the idea of A and B both have clear benefits and
> costs...
> > would like to see the opinions of folks --- do we  lean in one direction
> or
> > another? What is the criteria for adding a new feature, package, stack to
> > bigtop?
> >
> > ... Or maybe im just overthinking it and should be spending this time
> > testing spark for 0.9 release....
>
> I'd love to know what other think, but for 0.9 I'd rather stay the course.
>
> Thanks,
> Roman.
>
> P.S. There are also market forces at play that may fundamentally change
> the focus of what we're all working on in the year or so.
>

Mime
View raw message