bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: What will the next generation of bigtop look like?
Date Tue, 23 Dec 2014 21:29:51 GMT
On Tue, Dec 23, 2014 at 08:55AM, Andrew Purtell wrote:
> > So, in this sense commercials aren't releasing Apache software, but
> rather its derivatives. I don't see how Bigtop would compete in this field
> neither culturally nor resource wise.
> 
> This. From personal experience, once you start patching Apache releases you
> kick off a snowball of curation that gets larger with every upstream commit
> on each project. The only way to stop the ball rolling is to periodically
> nuke it with a rebase on a Apache release, reset that patch delta on as
> many components as possible to near 0. In practice Bigtop can't do this
> because we don't have the bandwidth. But in principle it is also good in my
> opinion: If the Apache releases cannot stand up a stable and cohesive stack
> then the ecosystem has issues, and Bigtop can provide a corrective
> influence as patches, JIRAs, email.

Same here. Although, when we were doing a Hadoop distribution at WANdisco it
was 100% Apache, e.g. Bigtop stack with a couple of our own components on top
of it. Otherwise, you risk a divergence or a huge effort to simply keep this
business of yours afloat.

Cos

> On Tue, Dec 23, 2014 at 12:36 AM, Konstantin Boudnik <cos@apache.org> wrote:
> >
> > Sorry for being a bit late to this discussion, but I will try to make a
> > very
> > sort summary of what I've read:
> >
> > 1. there's the intention to focus on the vertical value-add,
> >    rather than just a platform [Andrew, et all]
> > 2. Focus more on in-memory technologies (which seemed to be our trend ever
> > since
> >    we added Spark and now Ignite (incubating)). [Jay, Evans, RJ]
> > 3. while many data processing components aren't HDFS centric anymore, the
> >    storage layer still seems to be important for anything related to
> > Hadoop.
> >    Since, I don't think HDFS can be dropped tomorrow.
> >
> > And I don't see these three interfering with each other. To me they are
> > quite
> > complemektary.
> >
> > As for lower appeal of Bigtop stack to commercials as Evans alluded.
> > There's
> > that. And the main reason is that Bigtop releases are always based on
> > official
> > Apache release of upstream components. Wheres non of the Hadoop vendors can
> > say the same. Anything that Cloudera or HortonWorks put out there is
> > Hadoop 2.x
> > + N-patches, where N could be anywhere between 1 and 2000, but is never 0.
> >
> > So, in this sense commercials aren't releasing Apache software, but rather
> > its
> > derivatives. I don't see how Bigtop would compete in this field neither
> > culturally nor resource wise.
> >
> > Cos
> >
> > On Sat, Dec 06, 2014 at 06:23PM, jay vyas wrote:
> > > hi bigtop !
> > >
> > > I thought id start a thread a few vaguely related thoughts i have around
> > > next couple iterations of bigtop.
> > >
> > > 1) Hive:  How will bigtop to evolve to support it, now that it is much
> > > more  than a mapreduce query wrapper?
> > >
> > > 2) I wonder wether we should confirm cassandra interoperability of spark
> > in
> > > bigtop distros,
> > >
> > > 3) Also, as per , https://issues.apache.org/jira/browse/BIGTOP-1561 ---
> > > What about presto ?  Who is interested in supporting it - packaging it
> > > -testing it etc..?  (con) I don't know if its really ready to be in
> > bigtop,
> > > but (pro) i think if there is someone really dedicated to testing its
> > > interop w/ the bigtop stack, that could be great news for us.
> > >
> > > *** Now three concrete questions lead to a more interesting question ***
> > >
> > > 4) in general, i think bigtop can move in one of 3 directions.
> > >
> > >   EXPAND ? : Expanding to include new components, with just basic
> > interop,
> > > and let folks evolve their own stacks on top of bigtop on their own.
> > >
> > >   CONTRACT+FOCUS ?  Contracting to focus on a lean set of core
> > components,
> > > with super high quality.
> > >
> > >   STAY THE COURSE ? Staying the same ~ a packaging platform for just
> > > hadoop's direct ecosystem.
> > >
> > > I am intrigued by the idea of A and B both have clear benefits and
> > > costs...  would like to see the opinions of folks --- do we  lean in one
> > > direction or another? What is the criteria for adding a new feature,
> > > package, stack to bigtop?
> > >
> > > ... Or maybe im just overthinking it and should be spending this time
> > > testing spark for 0.9 release....
> > >
> > > Either way, looking forward to some feedback on these thoughts from the
> > > bigtop community !
> > >
> > > jay vyas
> >
> 
> 
> -- 
> Best regards,
> 
>    - Andy
> 
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Mime
View raw message