bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <>
Subject Re: What will the next generation of bigtop look like?
Date Tue, 23 Dec 2014 08:36:20 GMT
Sorry for being a bit late to this discussion, but I will try to make a very
sort summary of what I've read:

1. there's the intention to focus on the vertical value-add,
   rather than just a platform [Andrew, et all]
2. Focus more on in-memory technologies (which seemed to be our trend ever since
   we added Spark and now Ignite (incubating)). [Jay, Evans, RJ]
3. while many data processing components aren't HDFS centric anymore, the
   storage layer still seems to be important for anything related to Hadoop.
   Since, I don't think HDFS can be dropped tomorrow.

And I don't see these three interfering with each other. To me they are quite

As for lower appeal of Bigtop stack to commercials as Evans alluded. There's
that. And the main reason is that Bigtop releases are always based on official
Apache release of upstream components. Wheres non of the Hadoop vendors can
say the same. Anything that Cloudera or HortonWorks put out there is Hadoop 2.x
+ N-patches, where N could be anywhere between 1 and 2000, but is never 0.

So, in this sense commercials aren't releasing Apache software, but rather its
derivatives. I don't see how Bigtop would compete in this field neither
culturally nor resource wise.


On Sat, Dec 06, 2014 at 06:23PM, jay vyas wrote:
> hi bigtop !
> I thought id start a thread a few vaguely related thoughts i have around
> next couple iterations of bigtop.
> 1) Hive:  How will bigtop to evolve to support it, now that it is much
> more  than a mapreduce query wrapper?
> 2) I wonder wether we should confirm cassandra interoperability of spark in
> bigtop distros,
> 3) Also, as per , ---
> What about presto ?  Who is interested in supporting it - packaging it
> -testing it etc..?  (con) I don't know if its really ready to be in bigtop,
> but (pro) i think if there is someone really dedicated to testing its
> interop w/ the bigtop stack, that could be great news for us.
> *** Now three concrete questions lead to a more interesting question ***
> 4) in general, i think bigtop can move in one of 3 directions.
>   EXPAND ? : Expanding to include new components, with just basic interop,
> and let folks evolve their own stacks on top of bigtop on their own.
>   CONTRACT+FOCUS ?  Contracting to focus on a lean set of core components,
> with super high quality.
>   STAY THE COURSE ? Staying the same ~ a packaging platform for just
> hadoop's direct ecosystem.
> I am intrigued by the idea of A and B both have clear benefits and
> costs...  would like to see the opinions of folks --- do we  lean in one
> direction or another? What is the criteria for adding a new feature,
> package, stack to bigtop?
> ... Or maybe im just overthinking it and should be spending this time
> testing spark for 0.9 release....
> Either way, looking forward to some feedback on these thoughts from the
> bigtop community !
> jay vyas

View raw message