incubator-bigtop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Mahé <>
Subject Re: Spark in-memory analytics in BigTop stack
Date Wed, 26 Sep 2012 08:19:46 GMT
zOn 09/25/2012 10:46 AM, MTG dev wrote:
> Hi there.
> Apparently, I am not in a position to say what role Spark can play in the
> Bigtop for I am not speaking for neither of those projects.
> However, I can tell that Spark provides a number of the advantages compare to
> a traditional MapReduce model: stateful computational model with a need to
> write everything back to file system after step, in-memory calculations,
> higher level of primitives expressed in a functional language, etc. These
> advantages combined with low-latency planner result in a very significant
> performance improvement. I'd suggest to go over for more
> information.
> I am not an expert on Drill, but I'd say that Spark give immediate benefits
> over the former because it is already here and can be used by anyone ;)
> As for integration with Bigtop: Spark doesn't require any special integration
> with the rest of the stack - it might use HDFS as the underlying storage, but
> that's about it.
> Looks like there's an ongoing development to allow Spark to use Hive's SerDes,
> but I am not completely sure about its status.
> On Mon, Sep 24, 2012 at 09:59PM, Roman Shaposhnik wrote:
>> On Mon, Sep 24, 2012 at 8:52 PM, Anatoli Fomenko <> wrote:
>>> Hi Alef,
>>> Great news!
>>> Spark developers are interested in developing Spark packages and
>>> contributing them to open source. Since you already have them,
>>> what would you think about contributing the source to BigTop?
> We don't have any plans of holding the sources of the packages back, but we
> are working on rpm packaging right now. Once the work is over, we should be
> able to contribute it back to the community. Shall there be a JIRA ticket for
> that or something?
> With regards,
>    Alef
>    MTG dev team

Great news!

And yes, there should be a ticket. It will be helpful to organize any 
work around it.


View raw message