hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raymond Jennings III <raymondj...@yahoo.com>
Subject Re: Is it possible to share a key across maps?
Date Sat, 09 Jan 2010 00:54:30 GMT
Hi, you do this in the map method (open the file and read the first line?)  Could you explain
a little more how you do it with configure(), thank you.

--- On Fri, 1/8/10, Gang Luo <lgpublic@yahoo.com.cn> wrote:

> From: Gang Luo <lgpublic@yahoo.com.cn>
> Subject: Re: Is it possible to share a key across maps?
> To: common-user@hadoop.apache.org
> Date: Friday, January 8, 2010, 4:46 PM
> I will do that like this: at each map
> task, I get the input file to
> this mapper in the configure(), and manually read the first
> line of
> that file to get the user ID. Then start running the map
> function.
> 
> 
> -Gang
> 
> 
> ----- 原始邮件 ----
> 发件人: Raymond Jennings III <raymondjiii@yahoo.com>
> 收件人: common-user@hadoop.apache.org
> 发送日期: 2010/1/8 (周五) 4:23:15 下午
> 主   题: Is it possible to share a key
> across maps?
> 
> I have large files where the userid is the first line of
> each file.  I want to use that value as the output of
> the map phase for each subsequent line of the file.  If
> each map task gets a chunk of this file only one map task
> will read the key value from the first line.  Is there
> anyway I can force the other map tasks to wait until this
> key is read and then somehow pass this value to other map
> tasks?  Or is my reasoning incorrect?  Thanks.
> 
> 
>      
> ___________________________________________________________
> 
>   好玩贺卡等你发,邮箱贺卡全新上线!
> 
> http://card.mail.cn.yahoo.com/
> 


      

Mime
View raw message