hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Mattas <anth...@mattas.net>
Subject Re: Impact of Tez/Spark to MapReduce
Date Thu, 06 Mar 2014 01:57:15 GMT
Unfortunately I’m not super familiar with Spark - I guess my curiosity stems from a deep
seated belief that big iron EDW type appliances are slowly going to fade out, so I’m trying
to really get my head around what that’s going to look like in the next few years.

Hive(Stinger)+Tez+Yarn seems very promising, Impala does as well but I’m not sure if the
more open Hive solution will be preferred longer term.  Does Map-Reduce still exist at that
time, or does it slowly fade away (I would assume its still around because there are a lot
of unique things you can do with MR today that isn’t easily accomplished in other frameworks).

On Mar 5, 2014, at 8:48 PM, Jeff Zhang <jezhang@gopivotal.com> wrote:

> I believe in the future the spark functional style api will dominate the big data world.
Very few people will use the native mapreduce API. Even now usually users use third-party
mapreduce library such as cascading, scalding, scoobi or script language hive, pig rather
than the native mapreduce api.  
> And this functional style of api compatible both with hadoop's mapreduce and spark's
RDD. The underlying execution engine will be transparent to users. So I guess or I hope in
the future, the api will be unified  while the underlying execution engine will been choose
intelligently according the resources you have and the metadata of the data you operate on.

> On Thu, Mar 6, 2014 at 9:02 AM, Edward Capriolo <edlinuxguru@gmail.com> wrote:
> The thing about yarn is you chose what is right for the the workload. 
> For example: Spark may not the right choice if for example join tables do not fit in
> On Wednesday, March 5, 2014, Anthony Mattas <anthony@mattas.net> wrote:
> > With Tez and Spark becoming mainstream what does Map Reduce look like longer term?
Will it become a component that sits on top of Tez, or will they continue to live side by
side utilizing YARN?
> > I'm struggling a little bit to understand what the roadmap looks like for the technologies
that sit on top of YARN.
> >
> > Anthony Mattas
> > anthony@mattas.net
> -- 
> Sorry this was sent from mobile. Will do less grammar and spell check than usual.

View raw message