hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shravan Matthur Narayanamurthy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-445) Null Pointer Exceptions in the mappers leading to lot of retries
Date Mon, 22 Sep 2008 20:25:47 GMT

     [ https://issues.apache.org/jira/browse/PIG-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Shravan Matthur Narayanamurthy updated PIG-445:

    Attachment: mq.patch

> Null Pointer Exceptions in the mappers leading to lot of retries
> ----------------------------------------------------------------
>                 Key: PIG-445
>                 URL: https://issues.apache.org/jira/browse/PIG-445
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>    Affects Versions: types_branch
>            Reporter: Shravan Matthur Narayanamurthy
>            Assignee: Shravan Matthur Narayanamurthy
>         Attachments: mq.patch
> Even with successfully completed jobs, usually with a large data set, we see that there
are NPE produced in the mappers which lead to task failure. However, this problem goes away
on retries. The problem occurs at places where we access the reporter to report progress.

> From the analysis, this should happen with jobs that use combiner. The combiner is called
whenever the mapper outputs a buffer full of data. So the combiner is called multiple times
in between a map task. In the Combiner.close method we currently set the reporter to null
as it was assumed that combiner is called only after the entire output of map is produced.

> The fix is to not set the reporter to null in the Combiner.close() method

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message