bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Rebooting the conversation on the Future of bigtop: Abstracting the backplane ? Containers?
Date Mon, 15 Jun 2015 16:29:02 GMT
> Is it time for us to pick a resource manager?

Not if we want to be like a Debian for big data software.  I'm not sure we
want to limit our reach by being overly opinionated. With my user's hat on,
if we don't package Hadoop and YARN, then I wouldn't have any use for
Bigtop.

> Nowadays folks don't necessarily need the whole hadoop ecosystem, and
there is a huge shift to in-memory, monolithic stacks happening

A Bigtop user would only need to install the packages they would like to
use, right? Is this an argument for exclusion? Exclusion of what?

> Is it time to start experimenting with running docker tarballs ?

T​his sounds fine as an additional target for builds, but not if it leads
to a proposal to do away with
​ the​
OS native packaging.
​That's useful too. Containers are trendy but not useful or even
appropriate for every environment or use case.​
​ ​




On Mon, Jun 15, 2015 at 9:22 AM, jay vyas <jayunit100.apache@gmail.com>
wrote:

> Hi folks.   Every few months, i try to reboot the conversation about the
> next generation of bigtop.
>
> There are 3 things which i think we should consider : A backplane (rather
> than deploy to machines, the meaning of the term "ecosystem" in a
> post-spark in-memory apacolypse, and containerization.
>
> 1) BACKPLANE: The new trend is to have a backplane that provides
> networking abstractions for you (mesos, kubernetes, yarn, and so on).   Is
> it time for us to pick a resource manager?
>
> 2) ECOSYSTEM?: Nowadays folks don't necessarily need the whole hadoop
> ecosystem, and there is a huge shift to in-memory, monolithic stacks
> happening (i.e. gridgain or spark can do what 90% of the hadoop ecosystem
> already does, supporting streams, batch,sql all in one).
>
> 3) CONTAINERS:  we are doing a great job w/ docker in our build infra.  Is
> it time to start experimenting with running docker tarballs ?
>
> Combining 1+2+3 - i could see a useful bigdata upstream distro which (1)
> just installed an HCFS implementation (gluster,HDFS,...) along side, say,
> (2) mesos as a backplane for the tooling for [[ hbase + spark + ignite ]]
> --- and then (3) do the integration testing of available mesos-framework
> plugins for ignite and spark underneath.  If other folks are interested,
> maybe we could create the "1x" or "in-memory" branch to start hacking on it
> sometime ?    Maybe even bring the flink guys in as well, as they are
> interested in bigtop packaging.
>
>
>
> --
> jay vyas
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
View raw message