hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Viraj Bhat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3827) Jobs with empty map-outputs and intermediate compression fail
Date Fri, 25 Jul 2008 18:41:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12617002#action_12617002
] 

Viraj Bhat commented on HADOOP-3827:
------------------------------------

Here are the output and error logs for the maps and reduces which can result from this bug
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logs of the output from killed map - "m_005937_0"  with zero input and output bytes to hdfs
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
attempt_200807242354_0001_m_005937_0: No outputs to promote from hdfs://ymachine.mydomain.com/myhome/dir/_temporary/_attempt_200807242354_0001_m_005937_0
2008-07-25 00:05:55,986 INFO org.apache.hadoop.mapred.TaskRunner: Task 'attempt_200807242354_0001_m_005937_0'
done.
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
Error on map-side
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
Too many fetch-failures
Too many fetch-failures
Too many fetch-failures

--------------------------------------------------------------------------------------------------------------------------------------------------------------------
Logs of the output from killed reduce "attempt_200807242354_0001_r_000001_0 " as a result
of  - map "m_005937_0"  providing zero output bytes to the reducers
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
2008-07-25 00:06:00,618 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling 2 bytes (2 raw
bytes) into RAM from attempt_200807242354_0001_m_005937_0
2008-07-25 00:06:00,618 INFO org.apache.hadoop.mapred.ReduceTask: Read 0 bytes from map-output
for attempt_200807242354_0001_m_005937_0
2008-07-25 00:06:00,618 WARN org.apache.hadoop.mapred.ReduceTask: attempt_200807242354_0001_r_000001_0
copy failed: attempt_200807242354_0001_m_005937_0 from mymachine.mydomain.com
2008-07-25 00:06:00,618 WARN org.apache.hadoop.mapred.ReduceTask: java.io.IOException: Incomplete
map output received for attempt_200807242354_0001_m_005937_0 from http://mymachine.mydomain.com:55279/mapOutput?job=job_200807242354_0001&map=attempt_200807242354_0001_m_005937_0&reduce=1
(0 instead of 2)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1248)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1093)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:983)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:932)
........
........
2008-07-25 00:06:37,696 INFO org.apache.hadoop.mapred.ReduceTask: Failed to fetch map-output
from attempt_200807242354_0001_m_005937_0 even after MAX_FETCH_RETRIES_PER_MAP retries...
 reporting to the JobTracker
--------------------------------------------------------------------------------------------------------------------------------------------------------------------

> Jobs with empty map-outputs and intermediate compression fail
> -------------------------------------------------------------
>
>                 Key: HADOOP-3827
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3827
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.18.0
>
>         Attachments: HADOOP-3827_0_20080724.patch
>
>
> The corner case where there are zero map-outputs doesn't pass the codec to the IFile.Writer
leading to un-compressed data and subsequently failure on the reduce when it tries to decompress
that data.
> The straight-forward fix is to pass the codec:
> {noformat}
>            Writer<K, V> writer = new Writer<K, V>(job, finalOut, 
> -                                                 keyClass, valClass, null);
> +                                                 keyClass, valClass, codec);
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message