hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 谭军 <tanjun_2...@163.com>
Subject Re:RE: RE: RE: Can reducer get parameters from mapper besides key and value?
Date Tue, 09 Aug 2011 14:31:50 GMT
Hi Subroto,
Just one more thing

Mapper:

As “wordcount” mapper consumes line by line; try to consume the hdfs file line by line.

The output of Mapper will tell a node is connected directly to which node.

 

Reducer:

Reducer will consume mapper output and will hold the output of node (Key) and all the adjacent
node(value)

The output of Mapper do can tell a node is connected directly to which node
But how can it tell a node is connected indirectly (second neighbor, neighbor's neighbor)
to which node?
If Reducer can tell a node is connected indirectly to which node, how?
To get the adjacent node (value) of a node (key), the database file (gragh) is needed.
I think that's the problem. 

--


Regards!

Jun Tan


At 2011-08-09 21:41:45,"Subroto Sanyal" <subrotosanyal@huawei.com> wrote:


Hi Jun,

 

I hope you want to find out the adjacent nodes of node in a graph.

 

I have just gone through the mapper code (not very thoroughly). I feel the implementation
first needs to define the responsibility of Mapper and Reducer.

I can see the Mapper is doing a file read operation and reading it line by line….??????
Is it mappers responsibility to read file?

Mapper implementation is not using the Key and Value???

 

Just a simple suggestion:

Mapper:

As “wordcount” mapper consumes line by line; try to consume the hdfs file line by line.

The output of Mapper will tell a node is connected directly to which node.

 

Reducer:

Reducer will consume mapper output and will hold the output of node (Key) and all the adjacent
node(value)

 

The solution provided may not be optimized.

 

Regards,
Subroto Sanyal

From:谭军 [mailto:tanjun_2525@163.com]
Sent: Tuesday, August 09, 2011 5:24 PM
To:mapreduce-user@hadoop.apache.org;subroto.sanyal@huawei.com
Subject: Re:RE: RE: Can reducer get parameters from mapper besides key and value?

 

Hi Subroto,

It's very kind of u to help me.

I really appreciate it.

Now I attach the source code of my program.

It's about find neighbors in a gragh.

Such as gragh " A---B-----C", we define node B is the first neighbor of node A while C is
the second

The program is to find the first and second neighbors of A.

That's the scenario.

Node A is in the key file and the gragh(s) is in the database file.

Key file is given by users. Each line of key file is a nodename like

A

B

C

Database file is in hdfs servers ( in hard disk). Each line is an edge with 2 nodes like

A  B  0.5

B  C  0.7

0.5 and 0.7 are weights of the edges.

I can get B according to the key A and get C according to B by mapreduce

But I want to get B and C according to the key A.

Now the program attached can finish my job, however I don't think it's good enough.

It's too time cost. It's not the program I wanted.

 

Maybe u can help me.

Thanks very much!

--

Regards!

Jun Tan


At 2011-08-09 19:16:17,"Subroto Sanyal" <subrotosanyal@huawei.com> wrote:



Hi Jun

 

I mean that if I get some strings in mapper and I want to use them in reducer.

But they are neither keys nor values.

As per my understanding, there is no such way to pass an arbitrary reference from Mapper to
Reducer.

The information written in Output from Mapper is available to Reducer.

Further more, I don’t feel it will be good idea to keep such dependency.

Please let me know more about your scenario…may be we/community can suggest some solution…

 

By the way, Can reducer get side files in cache?

Please let me know about “Side Files”…..

 

Regards,
Subroto Sanyal

From:谭军 [mailto:tanjun_2525@163.com]
Sent: Tuesday, August 09, 2011 12:25 PM
To:mapreduce-user@hadoop.apache.org;subroto.sanyal@huawei.com
Subject: Re:RE: Can reducer get parameters from mapper besides key and value?

 

Hi Subroto,

I mean that if I get some strings in mapper and I want to use them in reducer.

But they are neither keys nor values.

 

By the way, Can reducer get side files in cache?

 

--

Regards!

Jun Tan


At 2011-08-09 14:42:10,"Subroto Sanyal" <subrotosanyal@huawei.com> wrote:




 

Hi Jun,

 

What is the file, list, string [] in this context?

I mean to say which file or list or string[].

 

The new MR (org.apache.hadoop.mapreduce.*) APIs has a parameter “context”. Request you
to browse through the APIs of Context (Inherited from JobContext->TaskAttemptContext).

The context parameter may provide you the reference you need.

 

Regards,
Subroto Sanyal

From:谭军 [mailto:tanjun_2525@163.com]
Sent: Tuesday, August 09, 2011 11:58 AM
To: mapreduce
Subject: Can reducer get parameters from mapper besides key and value?

 

Hi,

Can reducer gets parameters from mapper besides key and value?

Such as files, lists, string[] etc.

Thanks





--

Regards!

Jun Tan

















Mime
View raw message