incubator-bigtop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wing Yew Poon <wyp...@cloudera.com>
Subject Re: Spark in-memory analytics in BigTop stack
Date Wed, 26 Sep 2012 23:06:37 GMT
Hi,
please correct me if I'm wrong, but I thought Spark runs on top of
Mesos. Does it not require Mesos to run?
- Wing Yew

On Wed, Sep 26, 2012 at 10:42 AM, MTG dev <dev@magnatempusgroup.net> wrote:
> Thanks Bruno!
>
> I have created  BIGTOP-715
>
> Cheers,
>   MTG dev
>
> On Wed, Sep 26, 2012 at 01:19AM, Bruno Mahé wrote:
>> zOn 09/25/2012 10:46 AM, MTG dev wrote:
>> >Hi there.
>> >
>> >Apparently, I am not in a position to say what role Spark can play in the
>> >Bigtop for I am not speaking for neither of those projects.
>> >
>> >However, I can tell that Spark provides a number of the advantages compare to
>> >a traditional MapReduce model: stateful computational model with a need to
>> >write everything back to file system after step, in-memory calculations,
>> >higher level of primitives expressed in a functional language, etc. These
>> >advantages combined with low-latency planner result in a very significant
>> >performance improvement. I'd suggest to go over spark-project.org for more
>> >information.
>> >
>> >I am not an expert on Drill, but I'd say that Spark give immediate benefits
>> >over the former because it is already here and can be used by anyone ;)
>> >
>> >As for integration with Bigtop: Spark doesn't require any special integration
>> >with the rest of the stack - it might use HDFS as the underlying storage, but
>> >that's about it.
>> >
>> >Looks like there's an ongoing development to allow Spark to use Hive's SerDes,
>> >but I am not completely sure about its status.
>> >
>> >On Mon, Sep 24, 2012 at 09:59PM, Roman Shaposhnik wrote:
>> >>On Mon, Sep 24, 2012 at 8:52 PM, Anatoli Fomenko <afomenko@yahoo.com>
wrote:
>> >>>Hi Alef,
>> >>>
>> >>>Great news!
>> >>>
>> >>>Spark developers are interested in developing Spark packages and
>> >>>contributing them to open source. Since you already have them,
>> >>>what would you think about contributing the source to BigTop?
>> >
>> >We don't have any plans of holding the sources of the packages back, but we
>> >are working on rpm packaging right now. Once the work is over, we should be
>> >able to contribute it back to the community. Shall there be a JIRA ticket for
>> >that or something?
>> >
>> >With regards,
>> >   Alef
>> >   MTG dev team
>> >
>>
>>
>> Great news!
>>
>>
>> And yes, there should be a ticket. It will be helpful to organize
>> any work around it.
>>
>> Thanks,
>> Bruno
>>
>>

Mime
View raw message