hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Espen Amble Kolstad (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1062) Checksum error in InMemoryFileSystem
Date Mon, 05 Mar 2007 15:55:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12478080
] 

Espen Amble Kolstad commented on HADOOP-1062:
---------------------------------------------

On 2 attempt I got this addititonal error:
2007-03-05 16:05:57,883 WARN  mapred.TaskRunner - task_0001_r_000005_1 Final merge of the
inmemory files threw an exception: org.apache.hadoop.fs.ChecksumException: Checksum error:
/trank/nutch-0.9-dev/filesystem/mapred/local/task_0001_r_000005_1/map_2.out at 16776192
        at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.verifySum(ChecksumFileSystem.java:250)
        at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:207)
        at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumFileSystem.java:163)
        at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:41)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:57)
        at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:91)
        at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1300)
        at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1363)
        at org.apache.hadoop.io.SequenceFile$Reader.nextRawValue(SequenceFile.java:1656)
        at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawValue(SequenceFile.java:2579)
        at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.next(SequenceFile.java:2351)
        at org.apache.hadoop.io.SequenceFile$Sorter.writeFile(SequenceFile.java:2226)
        at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(ReduceTaskRunner.java:820)


> Checksum error in InMemoryFileSystem
> ------------------------------------
>
>                 Key: HADOOP-1062
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1062
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.12.0, 0.12.1
>            Reporter: Espen Amble Kolstad
>
> Getting the following error in the tasktracker log on 2 attempts:
> 2007-03-05 14:59:50,320 WARN  mapred.TaskRunner - task_0001_r_000005_0 Intermediate Merge
of the inmemory files threw an exception: org.apache.hadoop.fs.ChecksumException: Checksum
error: /trank/n
> utch-0.9-dev/filesystem/mapred/local/task_0001_r_000005_0/map_2.out at 16776192
>         at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.verifySum(ChecksumFileSystem.java:250)
>         at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.readBuffer(ChecksumFileSystem.java:207)
>         at org.apache.hadoop.fs.ChecksumFileSystem$FSInputChecker.read(ChecksumFileSystem.java:163)
>         at org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:41)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>         at java.io.DataInputStream.readFully(DataInputStream.java:178)
>         at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:57)
>         at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:91)
>         at org.apache.hadoop.io.SequenceFile$Reader.readBuffer(SequenceFile.java:1300)
>         at org.apache.hadoop.io.SequenceFile$Reader.seekToCurrentValue(SequenceFile.java:1363)
>         at org.apache.hadoop.io.SequenceFile$Reader.nextRawValue(SequenceFile.java:1656)
>         at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawValue(SequenceFile.java:2579)
>         at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.next(SequenceFile.java:2351)
>         at org.apache.hadoop.io.SequenceFile$Sorter.writeFile(SequenceFile.java:2226)
>         at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(ReduceTaskRunner.java:820)
> When I changed fs.inmemory.size.mb to 0 (was 75 - default) the reduce completes successfully.
> Could it be related to HADOOP-1027 or HADOOP-1014?
> - Espen

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message