Mailing-List: contact user-help@bigtop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@bigtop.apache.org
MIME-Version: 1.0
Sender: shaposhnik@gmail.com
In-Reply-To: 
 <CACVCA=ddu-z2i9p6wYSNMN1g3L+J7tM9cuxfpAk0G3TmEmMvNQ@mail.gmail.com>
References: 
 <CACVCA=ddu-z2i9p6wYSNMN1g3L+J7tM9cuxfpAk0G3TmEmMvNQ@mail.gmail.com>
Date: Mon, 15 Jun 2015 14:06:23 -0700
Message-ID: 
 <CA+ULb+v-BMB9vQCzhi-0gpxnuXcHNQj_2vEHkw8chqg1nWvUZQ@mail.gmail.com>
Subject: Re: Rebooting the conversation on the Future of bigtop: Abstracting
 the backplane ? Containers?
From: Roman Shaposhnik <roman@shaposhnik.org>
To: "user@bigtop.apache.org" <user@bigtop.apache.org>
Cc: "dev@bigtop.apache.org" <dev@bigtop.apache.org>
Content-Type: text/plain; charset=UTF-8

On Mon, Jun 15, 2015 at 9:22 AM, jay vyas <jayunit100.apache@gmail.com> wrote:
> Hi folks.   Every few months, i try to reboot the conversation about the
> next generation of bigtop.
>
> There are 3 things which i think we should consider : A backplane (rather
> than deploy to machines, the meaning of the term "ecosystem" in a post-spark
> in-memory apacolypse, and containerization.
>
> 1) BACKPLANE: The new trend is to have a backplane that provides networking
> abstractions for you (mesos, kubernetes, yarn, and so on).   Is it time for
> us to pick a resource manager?

Let me rephrase the above and see if we're talking about the same thing. To
me your question is really about "what does a datacenter look like to Bigtop".
Today a datacenter looks to Bigtop as a bunch of individual nodes running
some kind of a Linux distribution. What you seem to be asking is that whether
it is time for us to embrace the vision of a datacenter that looks like mesos,
etc. Correct?

Also, I don't think you're suggesting that we drop the bread-n-butter of Bigtop,
but I still need to make sure.

> 2) ECOSYSTEM?: Nowadays folks don't necessarily need the whole hadoop
> ecosystem, and there is a huge shift to in-memory, monolithic stacks
> happening (i.e. gridgain or spark can do what 90% of the hadoop ecosystem
> already does, supporting streams, batch,sql all in one).

Correct. That said, I'm not sure what it means for Bigtop.

> 3) CONTAINERS:  we are doing a great job w/ docker in our build infra.  Is
> it time to start experimenting with running docker tarballs ?

I think it is time, but

> Combining 1+2+3 - i could see a useful bigdata upstream distro which (1)
> just installed an HCFS implementation (gluster,HDFS,...) along side, say,
> (2) mesos as a backplane for the tooling for [[ hbase + spark + ignite ]]
> --- and then (3) do the integration testing of available mesos-framework
> plugins for ignite and spark underneath.  If other folks are interested,
> maybe we could create the "1x" or "in-memory" branch to start hacking on it
> sometime ?    Maybe even bring the flink guys in as well, as they are
> interested in bigtop packaging.

I'm actually very curious about use cases that folks might have around
traditional Hadoop Distributions. What you're articulating above seems
like one of those use cases, but at this point I'm sort of lost as to
what's the most common use case.

Thanks,
Roman.