hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 谭军 <tanjun_2...@163.com>
Subject Re:Re: Can I define a datastructure that all Mappers share?
Date Mon, 15 Aug 2011 11:29:10 GMT
Hi,
I want to construct a cross-linklist to store a graph
I want to get (node, neighbor-list) from (node, node) records in database file.
Each Mapper processes one node.
And each Mapper gets (node, node) records and passes (node, neighbor-list) records to Reducer.
How can I make it?

--


Regards!

Jun Tan


At 2011-08-15 18:25:06,"Sudharsan Sampath" <sudhan65@gmail.com> wrote:
Hi,


To my knowledge, its not possible with plain map-reduce. But you can try using a distributed
cache on top of it. To quote a few try, hazelcast (if ur prog lang is java) or gigaspace.


Just a note, why would you want to share date across mappers. It defeats the basic assumption
of map-reduce that mappers are independent work chunks.


-Sudhan S


2011/8/15 谭军<tanjun_2525@163.com>

Hi,
I want to define a matrix or list that all mappers share.
So that all mappers can do operations on it.
Can I make it?
Thanks!



--


Regards!

Jun Tan





Mime
View raw message