Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Arun C Murthy <acm@hortonworks.com>
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-464--384856943
Subject: Re: Checksum Error during Reduce Phase hadoop-1.0.2
Date: Thu, 16 Aug 2012 11:34:06 -0700
In-Reply-To: 
 <CAPxqZufdea=6cHLX8y0w3tW-TrEWcpH-7-+xHyPTUJEnv_8=sg@mail.gmail.com>
To: common-dev@hadoop.apache.org
References: 
 <CAPxqZufPB4V=XKKmd8KhHVHVuifTBqmJDRxvK6ivy6ALTgUEtg@mail.gmail.com>
 <CAOcnVr1Kjf7_xH9z-UsTkQeO1or7RvvfhGRvbmrfBffRHnGpwg@mail.gmail.com>
 <CAPxqZufdea=6cHLX8y0w3tW-TrEWcpH-7-+xHyPTUJEnv_8=sg@mail.gmail.com>
Message-Id: <F7FF69AF-2837-4034-B086-7F23AFCE993E@hortonworks.com>

--Apple-Mail-464--384856943
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Primarily, it could be caused by a corrupt disk - which is why checking =
if it's happening on a specific node(s) can help.

Arun

On Aug 16, 2012, at 10:04 AM, Pavan Kulkarni wrote:

> Harsh,
>=20
> I see this on couple of nodes.But what may be the cause of this error =
?Any
> idea about it? Thanks
>=20
> On Sun, Aug 12, 2012 at 9:06 AM, Harsh J <harsh@cloudera.com> wrote:
>=20
>> Hi Pavan,
>>=20
>> Do you see this happen on a specific node every time (i.e. when the
>> reducer runs there)?
>>=20
>> On Fri, Aug 10, 2012 at 11:43 PM, Pavan Kulkarni
>> <pavan.baburao@gmail.com> wrote:
>>> Hi,
>>>=20
>>> I am running a Terasort with a cluster of 8 nodes.The map phase
>> completes
>>> but when the reduce phase is around 68-70% I get this following =
error.
>>>=20
>>> *
>>> 12/08/10 11:02:36 INFO mapred.JobClient: Task Id :
>>> attempt_201208101018_0001_r_000027_0, Status : FAILED
>>> java.lang.RuntimeException: problem advancing post rec#38320220
>>> *
>>> *        at
>>> org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1214)*
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(Reduce=
Task.java:249)
>>> *
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.j=
ava:245)
>>> *
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.lib.IdentityReducer.reduce(IdentityReducer.java:4=
0)
>>> *
>>> *        at
>>> =
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)*
>>> *        at =
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)*
>>> *        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)*
>>> *        at java.security.AccessController.doPrivileged(Native =
Method)*
>>> *        at javax.security.auth.Subject.doAs(Subject.java:416)*
>>> *        at
>>>=20
>> =
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.=
java:1093)
>>> *
>>> *        at org.apache.hadoop.mapred.Child.main(Child.java:249)*
>>> *Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error*
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:164=
)*
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:101)*=

>>> *        at
>> org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:328)*
>>> *        at
>> org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java:358)*
>>> *        at
>>> org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:342)*
>>> *        at =
org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:374)*
>>> *        at
>> org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:220)*
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java=
:330)
>>> *
>>> *        at
>> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
>>> *
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$RawKVIteratorReader.next(=
ReduceTask.java:2531)
>>> *
>>> *        at
>> org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:220)*
>>> *        at
>>>=20
>> =
org.apache.hadoop.mapred.Merger$MergeQueue.adjustPriorityQueue(Merger.java=
:330)
>>> *
>>> *        at
>> org.apache.hadoop.mapred.Merger$MergeQueue.next(Merger.java:350)
>>> *
>>> *        at
>>> =
org.apache.hadoop.mapred.Task$ValuesIterator.readNextKey(Task.java:1253)*
>>> *        at
>>> org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1212)*
>>> *        ... 10 more*
>>>=20
>>> I came across somone facing the same
>>> issue<
>> =
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201001.mbox/%3=
C1c802db51001280427j5b8e57dai4a8d0fdd038f41@mail.gmail.com%3E
>>> in
>>> the mail-archives and he seemed to resolve it by listing hostnames =
in
>>> the */etc/hosts *file,
>>> but all my nodes have correct info about the hostnames in =
/etc/hosts,
>> but I
>>> still have these reducers throwing error.
>>> Any help regarding this issue is appreciated .Thanks
>>>=20
>>> --
>>>=20
>>> --With Regards
>>> Pavan Kulkarni
>>=20
>>=20
>>=20
>> --
>> Harsh J
>>=20
>=20
>=20
>=20
> --=20
>=20
> --With Regards
> Pavan Kulkarni

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


--Apple-Mail-464--384856943--