hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amogh Vasekar <am...@yahoo-inc.com>
Subject Re: Re: Re: Re: map output not euqal to reduce input
Date Fri, 11 Dec 2009 07:55:12 GMT
The counters are updated as the records are *consumed*, for both mapper and reducer. Can you
confirm if all the values returned by your iterators are consumed on reduce side? Also, are
you having feature of skipping bad records switched on?


On 12/11/09 4:32 AM, "Gang Luo" <lgpublic@yahoo.com.cn> wrote:

In the mapper of this job, I get something I am interested in for each
line and then output all of them. So the number of map input records is
equal to the map output records. Actually, I am doing semi join in this
job. There is no failure during execution.


----- ԭʼ�ʼ� ----
�����ˣ� Todd Lipcon <todd@cloudera.com>
�ռ��ˣ� common-user@hadoop.apache.org
�������ڣ� 2009/12/10 (����) 4:43:52 ����
��   �⣺ Re: Re�� Re�� map output not euqal to reduce input

On Thu, Dec 10, 2009 at 1:15 PM, Gang Luo <lgpublic@yahoo.com.cn> wrote:
> Hi Todd,
> I didn't change the partitioner, just use the default one. Will the default partitioner
cause the lost of the records?
> -Gang

Do the maps output data nondeterministically? Did you experience any
task failures in the run of the job?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message