Mailing-List: contact bigtop-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: bigtop-dev@incubator.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Date: Wed, 26 Sep 2012 16:16:40 -0700
From: Konstantin Boudnik <cos@apache.org>
To: bigtop-dev@incubator.apache.org
Subject: Re: Spark in-memory analytics in BigTop stack
Message-ID: <20120926231640.GK17164@linspire.com>
References: <505D3C97.3000306@apache.org> <20120923070635.GC18432@tpx>
 <1348439178.66635.YahooMailNeo@web161305.mail.bf1.yahoo.com>
 <20120924163121.GB7913@tpx>
 <1348545130.45699.YahooMailNeo@web161303.mail.bf1.yahoo.com>
 <CA+ULb+sFEy+TU7aTdv+J2XGCb2dhbfGhLkN+cOu-vZNT+2jj=A@mail.gmail.com>
 <20120925174613.GA22115@tpx> <5062BAA2.9060409@apache.org>
 <20120926174223.GA1449@tpx>
 <CADpuDdqxDzEGNeJP-p_ECbajUeU18cHBrD2M3LSvFQOcdVcdMQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: 
 <CADpuDdqxDzEGNeJP-p_ECbajUeU18cHBrD2M3LSvFQOcdVcdMQ@mail.gmail.com>
User-Agent: Mutt/1.5.18 (2008-05-17)

On Wed, Sep 26, 2012 at 04:06PM, Wing Yew Poon wrote:
> Hi,
> please correct me if I'm wrong, but I thought Spark runs on top of
> Mesos. Does it not require Mesos to run?

Yes, you are right. That's why Mesos library is a part of Spark's dependencies.

Now, there are 'on top of Mesos' and 'on Mesos cluster', which are quite
different apparently ;)

Cos

> - Wing Yew
> 
> On Wed, Sep 26, 2012 at 10:42 AM, MTG dev <dev@magnatempusgroup.net> wrote:
> > Thanks Bruno!
> >
> > I have created  BIGTOP-715
> >
> > Cheers,
> >   MTG dev
> >
> > On Wed, Sep 26, 2012 at 01:19AM, Bruno MahИ wrote:
> >> zOn 09/25/2012 10:46 AM, MTG dev wrote:
> >> >Hi there.
> >> >
> >> >Apparently, I am not in a position to say what role Spark can play in the
> >> >Bigtop for I am not speaking for neither of those projects.
> >> >
> >> >However, I can tell that Spark provides a number of the advantages compare to
> >> >a traditional MapReduce model: stateful computational model with a need to
> >> >write everything back to file system after step, in-memory calculations,
> >> >higher level of primitives expressed in a functional language, etc. These
> >> >advantages combined with low-latency planner result in a very significant
> >> >performance improvement. I'd suggest to go over spark-project.org for more
> >> >information.
> >> >
> >> >I am not an expert on Drill, but I'd say that Spark give immediate benefits
> >> >over the former because it is already here and can be used by anyone ;)
> >> >
> >> >As for integration with Bigtop: Spark doesn't require any special integration
> >> >with the rest of the stack - it might use HDFS as the underlying storage, but
> >> >that's about it.
> >> >
> >> >Looks like there's an ongoing development to allow Spark to use Hive's SerDes,
> >> >but I am not completely sure about its status.
> >> >
> >> >On Mon, Sep 24, 2012 at 09:59PM, Roman Shaposhnik wrote:
> >> >>On Mon, Sep 24, 2012 at 8:52 PM, Anatoli Fomenko <afomenko@yahoo.com> wrote:
> >> >>>Hi Alef,
> >> >>>
> >> >>>Great news!
> >> >>>
> >> >>>Spark developers are interested in developing Spark packages and
> >> >>>contributing them to open source. Since you already have them,
> >> >>>what would you think about contributing the source to BigTop?
> >> >
> >> >We don't have any plans of holding the sources of the packages back, but we
> >> >are working on rpm packaging right now. Once the work is over, we should be
> >> >able to contribute it back to the community. Shall there be a JIRA ticket for
> >> >that or something?
> >> >
> >> >With regards,
> >> >   Alef
> >> >   MTG dev team
> >> >
> >>
> >>
> >> Great news!
> >>
> >>
> >> And yes, there should be a ticket. It will be helpful to organize
> >> any work around it.
> >>
> >> Thanks,
> >> Bruno
> >>
> >>