hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Dahiya (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-728) Map-reduce task does not produce correct results when -reducer NONE is specified through streaming
Date Thu, 23 Nov 2006 22:17:05 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-728?page=comments#action_12452337 ] 
            
Sanjay Dahiya commented on HADOOP-728:
--------------------------------------

In case -reducer NONE is specified, useSingleSideOutputURI_ flag is true ( PipeMapred.java:332),
causing the value given in -output <path> to be treated as the output file for map task
[PipeMapRed.java:261]. 

It probably assumes that in reducer NONE, there will be no reduce output generated in DFS
and map tasks writing to DFS use the same assumption as the code for executing external commands
lies in configure(). 

The change I plan to make is to treat <path> as a directory name in case of -reduce
NONE, and use a unique suffix as in PipeMapRed.java:268, which includes task id in map output
file.

> Map-reduce task does not produce correct results when -reducer NONE is specified through
streaming
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-728
>                 URL: http://issues.apache.org/jira/browse/HADOOP-728
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/streaming
>            Reporter: dhruba borthakur
>         Assigned To: Sanjay Dahiya
>
> a) a file is create for the output instead of a directory.
> b) there is no way to understand what is going on from the client output
> I can produce an example for you, if you like -- but the behavior is consistent, so $HSTREAM
-mapper /bin/cat -reducer NONE should show the problem
> ~

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message