hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-1813) NPE in PipeMapred.MRErrorThread
Date Fri, 28 May 2010 06:21:37 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravi Gummadi updated MAPREDUCE-1813:
------------------------------------

    Attachment: 1813.patch

Attaching patch for trunk fixing the issue.

This patch sets the reporter of MRErrorThread(in PipeMapRed.java) to a dummy reporter at the
beginning of MRErrorThread.run() method just to avoid NPE. The actual reporter will be later
set properly as usual and reports as needed. In the interim(before the actual reporter is
set), the dummy reporter will just ignore the reporting lines.

This is the way MRErrorThread's reporter is set to dummy reporter in waitOutputThreads() for
the case of empty input based reducer. See HADOOP-4620 for details of the same.
But before control goes to waitOutputThreads(), if the MRErrorThread's reporter is accessed,
we get NPE. So to avoid this NPE, am setting this dummy reporter.

This patch adds testcases which tests the perl script given in earlier comment as mapper and
as reducer. Both test cases (1) perl script as streaming job's mapper and (2) perl script
as streaming based reducer
 ---- fail without the fix and pass with the fix.

Please review and provide your comments.

> NPE in PipeMapred.MRErrorThread
> -------------------------------
>
>                 Key: MAPREDUCE-1813
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1813
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/streaming
>    Affects Versions: 0.20.1
>            Reporter: Amareshwari Sriramadasu
>            Assignee: Ravi Gummadi
>             Fix For: 0.20.3
>
>         Attachments: 1813.patch
>
>
> Some reduce tasks fail with following NPE
> java.lang.RuntimeException: java.lang.NullPointerException
>         at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
>         at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:540)
>         at org.apache.hadoop.streaming.PipeReducer.close(PipeReducer.java:137)
>         at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:474)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412)
>         at org.apache.hadoop.mapred.Child.main(Child.java:159)
> Caused by: java.lang.NullPointerException
>        at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.setStatus(PipeMapRed.java:517)
>         at org.apache.hadoop.streaming.PipeMapRed$MRErrorThread.run(PipeMapRed.java:449)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message