hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-18) Under load the shuffle sometimes gets incorrect data
Date Thu, 16 Jul 2009 06:11:14 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-18?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravi Gummadi updated MAPREDUCE-18:
----------------------------------

        Fix Version/s: 0.21.0
    Affects Version/s: 0.21.0
         Release Note: This patch adds the mapid and reduceid in the http header of mapoutput
when being sent to reduce node. Also validates compressed length, decompressed length, mapid
and reduceid from http header at reduce node.
               Status: Patch Available  (was: Open)

> Under load the shuffle sometimes gets incorrect data
> ----------------------------------------------------
>
>                 Key: MAPREDUCE-18
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-18
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Owen O'Malley
>            Assignee: Ravi Gummadi
>            Priority: Blocker
>             Fix For: 0.21.0
>
>         Attachments: MR-18.patch, MR-18.v1.patch
>
>
> While testing HADOOP-5223 under load, we found reduces receiving completely incorrect
data. It was often random, but sometimes was the output of the wrong map for the wrong map.
It appears to either be a Jetty or JVM bug, but it is clearly happening on the server side.
In the HADOOP-5223 code, I added information about the map and reduce that were included and
we should add similar protection to 0.20 and trunk.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message