bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit100.apa...@gmail.com>
Subject Re: Rebooting the conversation on the Future of bigtop: Abstracting the backplane ? Containers?
Date Tue, 16 Jun 2015 12:03:04 GMT
Your right Bruno , I could, but I have no need of such a thing:)....  

And in any case --- this thread is just about sharing ideas, letting the whole community speak
up about their opinions on the future of bigtop. it's not about driving a particular project
direction. 

Bigtop is a unique project in that we integrate a lot of tools in a rapidly changing landscape,
so it's good to have some feelers out there to see what our users are thinking.  

Thanks all for the feedback, hope to get more!
 
> On Jun 16, 2015, at 2:11 AM, Bruno Mahé <bmahe@apache.org> wrote:
> 
>> On 06/15/2015 09:22 AM, jay vyas wrote:
>> Hi folks.   Every few months, i try to reboot the conversation about the next generation
of bigtop.
>> 
>> There are 3 things which i think we should consider : A backplane (rather than deploy
to machines, the meaning of the term "ecosystem" in a post-spark in-memory apacolypse, and
containerization.
>> 
>> 1) BACKPLANE: The new trend is to have a backplane that provides networking abstractions
for you (mesos, kubernetes, yarn, and so on).   Is it time for us to pick a resource manager?
>> 
>> 2) ECOSYSTEM?: Nowadays folks don't necessarily need the whole hadoop ecosystem,
and there is a huge shift to in-memory, monolithic stacks happening (i.e. gridgain or spark
can do what 90% of the hadoop ecosystem already does, supporting streams, batch,sql all in
one).
>> 
>> 3) CONTAINERS:  we are doing a great job w/ docker in our build infra.  Is it time
to start experimenting with running docker tarballs ?
>> 
>> Combining 1+2+3 - i could see a useful bigdata upstream distro which (1) just installed
an HCFS implementation (gluster,HDFS,...) along side, say, (2) mesos as a backplane for the
tooling for [[ hbase + spark + ignite ]] --- and then (3) do the integration testing of available
mesos-framework plugins for ignite and spark underneath.  If other folks are interested, maybe
we could create the "1x" or "in-memory" branch to start hacking on it sometime ?    Maybe
even bring the flink guys in as well, as they are interested in bigtop packaging.
>> 
>> 
>> 
>> -- 
>> jay vyas
> 
> 
> I have roughly the same position as Andrew on that matter.
> 
> What prevents you from starting something yourself to start hacking on it?
> 
> 
> Thanks,
> Bruno

Mime
View raw message