hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei Xue <simonxu...@gmail.com>
Subject Re: Is continuous map reduce supported
Date Fri, 03 Sep 2010 06:01:51 GMT
Thanks Jeff. Those are all valuable links.  It seems there are quite a few
people out there working on incremental MapReduce.

2010/9/1 Jeff Hammerbacher <hammer@cloudera.com>

> Hey Stephen,
>
> There have been several proposals for implementing such a feature. See
> https://issues.apache.org/jira/browse/MAPREDUCE-1211 for an implementation
> from Berkeley, now maintained at http://code.google.com/p/hop. The paper
> at https://www.ideals.illinois.edu/handle/2142/14819 describes a similar
> approach.
>
> Incremental bulk processing is another approach. See
> http://doi.acm.org/10.1145/1807128.1807138 for a system built on top of
> Hadoop, and http://research.microsoft.com/apps/pubs/default.aspx?id=117830for a system
built on top of Dryad.
>
> The blog post at http://clue.cs.washington.edu/node/14 describes a paper
> accepted at VLDB this year which improves the performance of Hadoop
> MapReduce for iterative tasks, and may be applicable to your research.
>
> Lastly, for more CEP-like approaches, you can check out C-MR from Brown (
> ftp://ftp.cs.brown.edu/pub/techreports/10/cs10-01.pdf) and Continuous
> MapReduce from UCSD: http://www.christrezzo.com/ctrezzo-thesis.pdf.
>
> As for actually being implemented in Hadoop MapReduce: the Apache project
> seems to have settled in to focus on stability rather than evolving new
> features.
>
> Thanks,
> Jeff
>
>
> On Tue, Aug 24, 2010 at 11:05 AM, Harsh J <qwertymaniac@gmail.com> wrote:
>
>> There's Chain-Mapping and Chain-Reducing available. With good docs:
>>
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/ChainReducer.html
>>
>> However, something as simple as Twister (which has iterative
>> mapreduces based on a while-like condition loop) isn't directly
>> available. One sometimes needs to chain jobs together to achieve this
>> in pure-Hadoop.
>>
>> Projects like Hive, Pig, and Cascading help with this a bit (plan
>> building, optimization of plan, execution, etc.).
>>
>> On Tue, Aug 24, 2010 at 10:25 PM, Stephen Mullins <smullins7@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > I have not used Hadoop but am researching it for an analytics project. I
>> > would like to know if Hadoop supports continuous or incremental map
>> reduce
>> > functionality. If not, are there any plans to add it?
>> >
>> > Thanks,
>> > Stephen
>> >
>>
>>
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>
>

Mime
View raw message