hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gang Luo <lgpub...@yahoo.com.cn>
Subject Re: map output not euqal to reduce input
Date Thu, 10 Dec 2009 16:31:47 GMT
No combiner. Here is the complete status.

 Launched reduce tasks=1
 Launched map tasks=3
Data-local map tasks=3

FILE_BYTES_READ=10266752
HDFS_BYTES_READ=18715130
 FILE_BYTES_WRITTEN=20533612
HDFS_BYTES_WRITTEN=250058

Reduce input groups=50000
Combine output records=0
Map input records=150000
Reduce shuffle bytes=10266764
Reduce output records=43282
Spilled Records=300000
Map output bytes=9966746
 Map input bytes=18711537
Combine input records=0
Map output records=150000
Reduce input records=93282

-Gang



----- 原始邮件 ----
发件人: Huy Phan <dachuy@gmail.com>
收件人: common-user@hadoop.apache.org
发送日期: 2009/12/10 (周四) 11:12:37 上午
主   题: Re: map output not euqal to reduce input

Do you have any combiner implemented in your job ?

On 12/10/2009 09:11 PM, Gang Luo wrote:
> Hi all,
> after finish one mapreduce job, the statistics shows that the number of records map generated
is not equal to the number of records that reduce input. It says:
>
> Map output records=150000
> Reduce input records=93282
>
> I think it is abnormal. Please give me some ideas how this happen and how to fix it.
Thanks.
>
>
> -Gang
>
>
>        ___________________________________________________________
>    潞脙脥忙潞脴驴篓碌脠脛茫路垄拢卢脫脢脧盲潞脴驴篓脠芦脨脗脡脧脧脽拢隆
> http://card.mail.cn.yahoo.com/
>
>    


      ___________________________________________________________ 
  好玩贺卡等你发,邮箱贺卡全新上线! 
http://card.mail.cn.yahoo.com/

Mime
View raw message