hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 谭军 <tanjun_2...@163.com>
Subject Re:Re:RE: RE: RE: Can reducer get parameters from mapper besides key and value?
Date Sun, 14 Aug 2011 13:25:37 GMT
Hi Subroto,
I cannot visit that video in China
Today, I got it somehow.
I got a question how can I make the number of Mappers and nodes the same?
There is no adjacent matrix of the graph in input files
How can I get the "seed" of the graph?

--


Regards!

Jun Tan




At 2011-08-10 02:51:08,"Subroto Sanyal 00900485" <subrotosanyal@huawei.com> wrote:
>Hi Jun,
>
>Basically, as per my understanding what you need is a graph traversal algorithm in MR
framework.
>Request you to go through this video tutorial(from Google). This provides clear illustration
of Graph Traversal Algorithm (BreadtFirstSearch) in MR.
>http://www.youtube.com/watch?v=BT-piFBP4fE&feature=player_embedded#at=812
>or
>http://www.danielblaisdell.com/2008/map-reduce-graph-traversal/
>
>Hope the mentioned links answers your questions.
>
>Regards,
>Subroto Sanyal
>
>Hi Subroto,
>Just one more thing
>Mapper:
>
>As “wordcount” mapper consumes line by line; try to consume the hdfs file line by
line.
>
>The output of Mapper will tell a node is connected directly to which node.
>
> 
>
>Reducer:
>
>Reducer will consume mapper output and will hold the output of node (Key) and all the
adjacent node(value)
>
>The output of Mapper do can tell a node is connected directly to which node
>But how can it tell a node is connected indirectly (second neighbor, neighbor's neighbor)
to which node?
>If Reducer can tell a node is connected indirectly to which node, how?
>To get the adjacent node (value) of a node (key), the database file (gragh) is needed.
>I think that's the problem. 
>
>--
>
>Regards!
>
>Jun Tan
>
>
>At 2011-08-09 21:41:45,"Subroto Sanyal" <subrotosanyal@huawei.com> wrote:
>
>Hi Jun,
>
> 
>
>I hope you want to find out the adjacent nodes of node in a graph.
>
> 
>
>I have just gone through the mapper code (not very thoroughly). I feel the implementation
first needs to define the responsibility of Mapper and Reducer.
>
>I can see the Mapper is doing a file read operation and reading it line by line….??????
Is it mappers responsibility to read file?
>
>Mapper implementation is not using the Key and Value???
>
> 
>
>Just a simple suggestion:
>
>Mapper:
>
>As “wordcount” mapper consumes line by line; try to consume the hdfs file line by
line.
>
>The output of Mapper will tell a node is connected directly to which node.
>
> 
>
>Reducer:
>
>Reducer will consume mapper output and will hold the output of node (Key) and all the
adjacent node(value)
>
> 
>
>The solution provided may not be optimized.
>
> 
>
>Regards, 
>Subroto Sanyal
>
>
>--------------------------------------------------------------------------------
>
>From: ?? [mailto:tanjun_2525@163.com] 
>Sent: Tuesday, August 09, 2011 5:24 PM
>To: mapreduce-user@hadoop.apache.org; subroto.sanyal@huawei.com
>Subject: Re:RE: RE: Can reducer get parameters from mapper besides key and value?
>
> 
>
>Hi Subroto,
>
>It's very kind of u to help me.
>
>I really appreciate it.
>
>Now I attach the source code of my program.
>
>It's about find neighbors in a gragh.
>
>Such as gragh " A---B-----C", we define node B is the first neighbor of node A while C
is the second
>
>The program is to find the first and second neighbors of A.
>
>That's the scenario.
>
>Node A is in the key file and the gragh(s) is in the database file.
>
>Key file is given by users. Each line of key file is a nodename like 
>
>A
>
>B
>
>C
>
>Database file is in hdfs servers ( in hard disk). Each line is an edge with 2 nodes like

>
>A  B  0.5
>
>B  C  0.7
>
>0.5 and 0.7 are weights of the edges.
>
>I can get B according to the key A and get C according to B by mapreduce
>
>But I want to get B and C according to the key A.
>
>Now the program attached can finish my job, however I don't think it's good enough.
>
>It's too time cost. It's not the program I wanted.
>
> 
>
>Maybe u can help me.
>
>Thanks very much!
>
>--
>
>Regards!
>
>Jun Tan
>
>
>At 2011-08-09 19:16:17,"Subroto Sanyal" <subrotosanyal@huawei.com> wrote:
>
>
>
>Hi Jun
>
> 
>
>I mean that if I get some strings in mapper and I want to use them in reducer.
>
>But they are neither keys nor values.
>
>As per my understanding, there is no such way to pass an arbitrary reference from Mapper
to Reducer.
>
>The information written in Output from Mapper is available to Reducer.
>
>Further more, I don’t feel it will be good idea to keep such dependency.
>
>Please let me know more about your scenario…may be we/community can suggest some solution…
>
> 
>
>By the way, Can reducer get side files in cache?
>
>Please let me know about “Side Files”…..
>
> 
>
>Regards, 
>Subroto Sanyal
>
>
>--------------------------------------------------------------------------------
>
>From: ?? [mailto:tanjun_2525@163.com] 
>Sent: Tuesday, August 09, 2011 12:25 PM
>To: mapreduce-user@hadoop.apache.org; subroto.sanyal@huawei.com
>Subject: Re:RE: Can reducer get parameters from mapper besides key and value?
>
> 
>
>Hi Subroto,
>
>I mean that if I get some strings in mapper and I want to use them in reducer.
>
>But they are neither keys nor values.
>
> 
>
>By the way, Can reducer get side files in cache?
>
> 
>
>--
>
>Regards!
>
>Jun Tan
>
>
>At 2011-08-09 14:42:10,"Subroto Sanyal" <subrotosanyal@huawei.com> wrote:
>
>
>
>
> 
>
>Hi Jun,
>
> 
>
>What is the file, list, string [] in this context?
>
>I mean to say which file or list or string[].
>
> 
>
>The new MR (org.apache.hadoop.mapreduce.*) APIs has a parameter “context”. Request
you to browse through the APIs of Context (Inherited from JobContext->TaskAttemptContext).
>
>The context parameter may provide you the reference you need.
>
> 
>
>Regards, 
>Subroto Sanyal
>
>
>--------------------------------------------------------------------------------
>
>From: ?? [mailto:tanjun_2525@163.com] 
>Sent: Tuesday, August 09, 2011 11:58 AM
>To: mapreduce
>Subject: Can reducer get parameters from mapper besides key and value?
>
> 
>
>Hi,
>
>Can reducer gets parameters from mapper besides key and value?
>
>Such as files, lists, string[] etc.
>
>Thanks
>
>
>
>
>
>--
>
>Regards!
>
>Jun Tan
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 
> 
>
Mime
View raw message