hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasad Pingali <pv...@research.iiit.ac.in>
Subject Re: Reduce task failed: org.apache.hadoop.fs.FSError: java.io.IOException
Date Fri, 12 Sep 2008 04:09:04 GMT
Thanks Arun,
   Yeah I think it was a disk problem. Redoing the task went through fine. 
Since disk corruptions may be common, doesn't hadoop pick a replicated block?

- Prasad.

On Thursday 11 September 2008 11:38:53 pm Arun C Murthy wrote:
> On Sep 11, 2008, at 9:10 AM, pvvpr wrote:
> > Hello,
> >  Never came across this error before. Upgraded to 0.18.0 this
> > morning and
> > ran a nutch fetch job. Got this exception in both the reduce
> > attempts of
> > a task and they failed. All other reducers seemed to work fine, except
> > for one task.
> >
> > Any ideas what could be the problem?
>
> Looks like data on disk got corrupted ( org.apache.hadoop.fs.FSError:
> java.io.IOException: Input/output error); could you check the disks?
>
> Arun
>
> > - Prasad Pingali.
> > IIIT, Hyderabad.
> >
> > 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0: Got 1 new map-outputs & number
> > of
> > known map outputs is 1
> > 2008-09-11 06:31:19,837 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Scheduled 1 of 1 known outputs (0
> > slow hosts and 0 dup hosts)
> > 2008-09-11 06:31:20,133 INFO org.apache.hadoop.mapred.ReduceTask:
> > Shuffling 8230686 bytes (8230686 raw bytes) into RAM from
> > attempt_200809101353_0021_m_000095_0
> > 2008-09-11 06:31:22,332 INFO org.apache.hadoop.mapred.ReduceTask: Read
> > 8230686 bytes from map-output for attempt_200809101353_0021_m_000095_0
> > 2008-09-11 06:31:22,333 INFO org.apache.hadoop.mapred.ReduceTask:
> > Rec #1
> > from attempt_200809101353_0021_m_000095_0 -> (33, 134) from machine10
> > 2008-09-11 06:31:28,837 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0: Got 1 new map-outputs & number
> > of
> > known map outputs is 1
> > 2008-09-11 06:31:28,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Scheduled 1 of 1 known outputs (0
> > slow hosts and 0 dup hosts)
> > 2008-09-11 06:31:29,585 INFO org.apache.hadoop.mapred.ReduceTask:
> > Shuffling 21454877 bytes (21454877 raw bytes) into Local-FS from
> > attempt_200809101353_0021_m_000074_0
> > 2008-09-11 06:31:37,831 INFO org.apache.hadoop.mapred.ReduceTask: Read
> > 21454877 bytes from map-output for
> > attempt_200809101353_0021_m_000074_0
> > 2008-09-11 06:31:37,832 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0We have  19 map outputs on disk.
> > Triggering merge of 10 files
> > 2008-09-11 06:31:38,033 INFO org.apache.hadoop.mapred.Merger:
> > Merging 10
> > sorted segments
> > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Need another 9 map output(s)
> > where 0
> > is already in progress
> > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0: Got 0 new map-outputs & number
> > of
> > known map outputs is 0
> > 2008-09-11 06:31:43,838 INFO org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Scheduled 0 of 0 known outputs (0
> > slow hosts and 0 dup hosts)
> > 2008-09-11 06:32:03,095 INFO org.apache.hadoop.mapred.Merger: Down
> > to the
> > last merge-pass, with 10 segments left of total size: 262561859 bytes
> > 2008-09-11 06:32:51,044 WARN org.apache.hadoop.mapred.ReduceTask:
> > attempt_200809101353_0021_r_000004_0 Merging of the local FS files
> > threw
> > an exception: org.apache.hadoop.fs.FSError: java.io.IOException:
> > Input/output error
> >        at
> > org.apache.hadoop.fs.RawLocalFileSystem
> > $LocalFSFileInputStream.read(RawLocalFileSystem.java:149)
> >        at java.io.BufferedInputStream.read1(BufferedInputStream.java:
> > 256)
> >        at java.io.BufferedInputStream.read(BufferedInputStream.java:
> > 317)
> >        at java.io.DataInputStream.read(DataInputStream.java:132)
> >        at
> > org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:380)
> >        at
> > org.apache.hadoop.fs.ChecksumFileSystem
> > $ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:208)
> >        at
> > org
> > .apache
> > .hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:236)
> >        at
> > org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:191)
> >        at
> > org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159)
> >        at java.io.DataInputStream.read(DataInputStream.java:132)
> >        at org.apache.hadoop.mapred.IFile$Reader.readData(IFile.java:
> > 263)
> >        at org.apache.hadoop.mapred.IFile$Reader.rejigData(IFile.java:
> > 293)
> >        at
> > org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:277)
> >        at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:339)
> >        at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:
> > 134)
> >        at
> > org.apache.hadoop.mapred.Merger
> > $MergeQueue.adjustPriorityQueue(Merger.java:225)
> >        at org.apache.hadoop.mapred.Merger
> > $MergeQueue.next(Merger.java:242)
> >        at org.apache.hadoop.mapred.Merger.writeFile(Merger.java:83)
> >        at
> > org.apache.hadoop.mapred.ReduceTask$ReduceCopier
> > $LocalFSMerger.run(ReduceTask.java:2021)
> > Caused by: java.io.IOException: Input/output error
> >        at java.io.FileInputStream.readBytes(Native Method)
> >        at java.io.FileInputStream.read(FileInputStream.java:199)
> >        at
> > org.apache.hadoop.fs.RawLocalFileSystem
> > $TrackingFileInputStream.read(RawLocalFileSystem.java:90)
> >        at
> > org.apache.hadoop.fs.RawLocalFileSystem
> > $LocalFSFileInputStream.read(RawLocalFileSystem.java:143)





Mime
View raw message