Return-Path: 
 <hadoop-dev-return-12692-apmail-lucene-hadoop-dev-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org
Received: (qmail 27874 invoked from network); 1 Jun 2007 20:33:39 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 1 Jun 2007 20:33:39 -0000
Received: (qmail 65659 invoked by uid 500); 1 Jun 2007 20:33:41 -0000
Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org
Received: (qmail 65629 invoked by uid 500); 1 Jun 2007 20:33:41 -0000
Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hadoop-dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:hadoop-dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:hadoop-dev@lucene.apache.org>
List-Id: <hadoop-dev.lucene.apache.org>
Reply-To: hadoop-dev@lucene.apache.org
Delivered-To: mailing list hadoop-dev@lucene.apache.org
Received: (qmail 65487 invoked by uid 99); 1 Jun 2007 20:33:41 -0000
Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jun 2007 13:33:40 -0700
X-ASF-Spam-Status: No, hits=-100.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Jun 2007 13:33:36 -0700
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 52B68714198
	for <hadoop-dev@lucene.apache.org>; Fri,  1 Jun 2007 13:33:16 -0700 (PDT)
Message-ID: <33133355.1180729996334.JavaMail.jira@brutus>
Date: Fri, 1 Jun 2007 13:33:16 -0700 (PDT)
From: "Owen O'Malley (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Updated: (HADOOP-1452) map output transfers of more than
 2^31 bytes output are failing
In-Reply-To: <29542170.1180729035583.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/HADOOP-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Owen O'Malley updated HADOOP-1452:
----------------------------------

    Fix Version/s: 0.13.0
         Assignee: Owen O'Malley
         Priority: Blocker  (was: Major)

> map output transfers of more than 2^31 bytes output are failing
> ---------------------------------------------------------------
>
>                 Key: HADOOP-1452
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1452
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.13.0
>            Reporter: Christian Kunz
>            Assignee: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.13.0
>
>
> Symptom:
> WARN org.apache.hadoop.mapred.ReduceTask: java.io.IOException: Incomplete map output received for http://<host>:50060/mapOutput?map=task_0026_m_000298_0&reduce=61 (2327458761 instead of 2327347307)
> WARN org.apache.hadoop.mapred.ReduceTask: task_0026_r_000061_0 adding host <host> to penalty box, next contact in 263 seconds
> Besides failing to fetch data, the reduce will retry forever. This should be limited.
> Source of the problem:
> in mapred/TaskTracker.java the variable totalRead keeping track what is sent to the reducer should be declared as long:
> ...
>         int totalRead = 0;
>         int len = mapOutputIn.read(buffer, 0,
>                                    partLength < MAX_BYTES_TO_READ
>                                    ? (int)partLength : MAX_BYTES_TO_READ);
>         while (len > 0) {
>           try {
>             outStream.write(buffer, 0, len);
>             outStream.flush();
>           } catch (IOException ie) {
>             isInputException = false;
>             throw ie;
>           }
>           totalRead += len;
>           if (totalRead == partLength) break;
>           len = mapOutputIn.read(buffer, 0,
>                                  (partLength - totalRead) < MAX_BYTES_TO_READ
>                                  ? (int)(partLength - totalRead) : MAX_BYTES_TO_READ);
>         }
> ...

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.