hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gang Luo <lgpub...@yahoo.com.cn>
Subject Re: Is it possible to share a key across maps?
Date Sat, 09 Jan 2010 03:03:01 GMT
I don't do that in map method, but in configure( JobConf ) method which runs ahead of any map
method call in that map task. JobConf.get("map.input.file") can tell you which file this map
task is processing. Use this path to read first line of corresponding file. All these are
done in configure method, that means, before any map method is called.


-Gang



----- 原始邮件 ----
发件人: Raymond Jennings III <raymondjiii@yahoo.com>
收件人: common-user@hadoop.apache.org
发送日期: 2010/1/8 (周五) 7:54:30 下午
主   题: Re: Is it possible to share a key across maps?

Hi, you do this in the map method (open the file and read the first line?)  Could you explain
a little more how you do it with configure(), thank you.

--- On Fri, 1/8/10, Gang Luo <lgpublic@yahoo.com.cn> wrote:

> From: Gang Luo <lgpublic@yahoo.com.cn>
> Subject: Re: Is it possible to share a key across maps?
> To: common-user@hadoop.apache.org
> Date: Friday, January 8, 2010, 4:46 PM
> I will do that like this: at each map
> task, I get the input file to
> this mapper in the configure(), and manually read the first
> line of
> that file to get the user ID. Then start running the map
> function.
> 
> 
> -Gang
> 
> 
> ----- 原始邮件 ----
> 发件人: Raymond Jennings III <raymondjiii@yahoo.com>
> 收件人: common-user@hadoop.apache.org
> 发送日期: 2010/1/8 (周五) 4:23:15 下午
> 主   题: Is it possible to share a key
> across maps?
> 
> I have large files where the userid is the first line of
> each file.  I want to use that value as the output of
> the map phase for each subsequent line of the file.  If
> each map task gets a chunk of this file only one map task
> will read the key value from the first line.  Is there
> anyway I can force the other map tasks to wait until this
> key is read and then somehow pass this value to other map
> tasks?  Or is my reasoning incorrect?  Thanks.
> 
> 
>      
> ___________________________________________________________
> 
>   好玩贺卡等你发,邮箱贺卡全新上线!
> 
> http://card.mail.cn.yahoo.com/
> 


      ___________________________________________________________ 
  好玩贺卡等你发,邮箱贺卡全新上线! 
http://card.mail.cn.yahoo.com/

Mime
View raw message