hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gang Luo <lgpub...@yahoo.com.cn>
Subject Re: Is it possible to share a key across maps?
Date Fri, 08 Jan 2010 21:46:22 GMT
I will do that like this: at each map task, I get the input file to
this mapper in the configure(), and manually read the first line of
that file to get the user ID. Then start running the map function.


----- 原始邮件 ----
发件人: Raymond Jennings III <raymondjiii@yahoo.com>
收件人: common-user@hadoop.apache.org
发送日期: 2010/1/8 (周五) 4:23:15 下午
主   题: Is it possible to share a key across maps?

I have large files where the userid is the first line of each file.  I want to use that value
as the output of the map phase for each subsequent line of the file.  If each map task gets
a chunk of this file only one map task will read the key value from the first line.  Is there
anyway I can force the other map tasks to wait until this key is read and then somehow pass
this value to other map tasks?  Or is my reasoning incorrect?  Thanks.


View raw message