hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Huang <jason.hu...@icare.com>
Subject Re: HDFS Error - BlockReader: error in packet header
Date Sun, 16 Sep 2012 21:55:56 GMT
I tried to reinstall hadoop and remove all the previous hdfs
directories and reformat the name node.

After that it appears to be working now.

thanks for looking at this!

Jason

On Sat, Sep 15, 2012 at 2:37 PM, Jason Huang <jason.huang@icare.com> wrote:
> Thanks Harsh.
>
> I've tried the following again:
> $ ./bin/hadoop jar hadoop-examples-1.0.3.jar pi 10 100
>
> And I got the same error (sorry for having to paste this longggg log):
> Number of Maps  = 10
> Samples per Map = 100
> Wrote input for Map #0
> Wrote input for Map #1
> Wrote input for Map #2
> Wrote input for Map #3
> Wrote input for Map #4
> Wrote input for Map #5
> Wrote input for Map #6
> Wrote input for Map #7
> Wrote input for Map #8
> Wrote input for Map #9
> Starting Job
> 12/09/15 14:20:02 INFO mapred.FileInputFormat: Total input paths to process : 10
> 12/09/15 14:20:02 INFO mapred.JobClient: Running job: job_201209151409_0001
> 12/09/15 14:20:03 INFO mapred.JobClient:  map 0% reduce 0%
> 12/09/15 14:20:14 INFO mapred.JobClient: Task Id :
> attempt_201209151409_0001_m_000011_0, Status : FAILED
> Error initializing attempt_201209151409_0001_m_000011_0:
> java.io.IOException: BlockReader: error in packet header(chunkOffset :
> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>         at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
>         at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
>         at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
>         at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
>         at java.io.DataInputStream.read(DataInputStream.java:100)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
>         at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
>         at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
>         at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
>         at java.lang.Thread.run(Thread.java:636)
>
> 12/09/15 14:20:14 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_0&filter=stdout
> 12/09/15 14:20:14 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_0&filter=stderr
> 12/09/15 14:20:23 INFO mapred.JobClient: Task Id :
> attempt_201209151409_0001_m_000011_1, Status : FAILED
> Error initializing attempt_201209151409_0001_m_000011_1:
> java.io.IOException: BlockReader: error in packet header(chunkOffset :
> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>         at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
>         at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
>         at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
>         at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
>         at java.io.DataInputStream.read(DataInputStream.java:100)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
>         at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
>         at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
>         at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
>         at java.lang.Thread.run(Thread.java:636)
>
> 12/09/15 14:20:23 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_1&filter=stdout
> 12/09/15 14:20:23 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_1&filter=stderr
> 12/09/15 14:20:32 INFO mapred.JobClient: Task Id :
> attempt_201209151409_0001_m_000011_2, Status : FAILED
> Error initializing attempt_201209151409_0001_m_000011_2:
> java.io.IOException: BlockReader: error in packet header(chunkOffset :
> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>         at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
>         at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
>         at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
>         at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
>         at java.io.DataInputStream.read(DataInputStream.java:100)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
>         at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
>         at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
>         at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
>         at java.lang.Thread.run(Thread.java:636)
>
> 12/09/15 14:20:32 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_2&filter=stdout
> 12/09/15 14:20:32 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_2&filter=stderr
> 12/09/15 14:20:50 INFO mapred.JobClient: Task Id :
> attempt_201209151409_0001_m_000010_0, Status : FAILED
> Error initializing attempt_201209151409_0001_m_000010_0:
> java.io.IOException: BlockReader: error in packet header(chunkOffset :
> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>         at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
>         at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
>         at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
>         at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
>         at java.io.DataInputStream.read(DataInputStream.java:100)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
>         at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
>         at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
>         at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
>         at java.lang.Thread.run(Thread.java:636)
>
> 12/09/15 14:20:50 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_0&filter=stdout
> 12/09/15 14:20:50 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_0&filter=stderr
> 12/09/15 14:20:59 INFO mapred.JobClient: Task Id :
> attempt_201209151409_0001_m_000010_1, Status : FAILED
> Error initializing attempt_201209151409_0001_m_000010_1:
> java.io.IOException: BlockReader: error in packet header(chunkOffset :
> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>         at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
>         at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
>         at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
>         at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
>         at java.io.DataInputStream.read(DataInputStream.java:100)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
>         at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
>         at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
>         at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
>         at java.lang.Thread.run(Thread.java:636)
>
> 12/09/15 14:20:59 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_1&filter=stdout
> 12/09/15 14:20:59 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_1&filter=stderr
> 12/09/15 14:21:08 INFO mapred.JobClient: Task Id :
> attempt_201209151409_0001_m_000010_2, Status : FAILED
> Error initializing attempt_201209151409_0001_m_000010_2:
> java.io.IOException: BlockReader: error in packet header(chunkOffset :
> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>         at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
>         at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
>         at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
>         at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
>         at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
>         at java.io.DataInputStream.read(DataInputStream.java:100)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
>         at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
>         at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
>         at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
>         at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
>         at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:416)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>         at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
>         at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
>         at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
>         at java.lang.Thread.run(Thread.java:636)
>
> 12/09/15 14:21:08 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_2&filter=stdout
> 12/09/15 14:21:08 WARN mapred.JobClient: Error reading task
> outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_2&filter=stderr
> 12/09/15 14:21:17 INFO mapred.JobClient: Job complete: job_201209151409_0001
> 12/09/15 14:21:17 INFO mapred.JobClient: Counters: 4
> 12/09/15 14:21:17 INFO mapred.JobClient:   Job Counters
> 12/09/15 14:21:17 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=0
> 12/09/15 14:21:17 INFO mapred.JobClient:     Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 12/09/15 14:21:17 INFO mapred.JobClient:     Total time spent by all
> maps waiting after reserving slots (ms)=0
> 12/09/15 14:21:17 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
> 12/09/15 14:21:17 INFO mapred.JobClient: Job Failed: JobCleanup Task
> Failure, Task: task_201209151409_0001_m_000010
> java.io.IOException: Job failed!
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
>         at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:297)
>         at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>         at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>         at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>
> Here's the a 'brief' version of the data node log (but still very long...):
> 2012-09-15 14:20:02,640 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_39584416619615086_1077 src: /127.0.0.1:49829 dest:
> /127.0.0.1:50010 2012-09-15 14:20:02,642 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:49829, dest: /127.0.0.1:50010, bytes: 20494, op:
> HDFS_WRITE, cliID: DFSClient_1015299679, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_39584416619615086_1077, duration: 565000 2012-09-15 14:20:02,642
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
> 0 for block blk_39584416619615086_1077 terminating 2012-09-15
> 14:20:02,663 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49831, bytes: 20658, op: HDFS_READ,
> cliID: DFSClient_672168163, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_39584416619615086_1077, duration: 270000 2012-09-15 14:20:02,665
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49832, bytes: 18, op: HDFS_READ,
> cliID: DFSClient_672168163, offset: 20480, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_39584416619615086_1077, duration: 146000 2012-09-15 14:20:02,757
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-6641115981657984283_1079 src: /127.0.0.1:49833 dest:
> /127.0.0.1:50010 2012-09-15 14:20:02,761 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:49833, dest: /127.0.0.1:50010, bytes: 20563, op:
> HDFS_WRITE, cliID: DFSClient_672168163, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_-6641115981657984283_1079, duration: 2189000 2012-09-15
> 14:20:02,761 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> PacketResponder 0 for block blk_-6641115981657984283_1079 terminating
> 2012-09-15 14:20:02,776 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_-6285781475981276067_1080 src: /127.0.0.1:49835 dest:
> /127.0.0.1:50010 2012-09-15 14:20:02,777 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:49835, dest: /127.0.0.1:50010, bytes: 106, op: HDFS_WRITE,
> cliID: DFSClient_1436299339, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_-6285781475981276067_1080, duration: 321000 2012-09-15
> 14:20:02,777 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> PacketResponder 0 for block blk_-6285781475981276067_1080 terminating
> 2012-09-15 14:20:02,781 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49836, bytes: 201, op: HDFS_READ,
> cliID: DFSClient_672168163, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_-6638660277808139598_1076, duration: 152000 2012-09-15
> 14:20:05,555 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49838, bytes: 110, op: HDFS_READ,
> cliID: DFSClient_1214970016, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_-6285781475981276067_1080, duration: 158000 2012-09-15
> 14:20:05,563 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49840, bytes: 20658, op: HDFS_READ,
> cliID: DFSClient_1762809953, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_39584416619615086_1077, duration: 302000  ................
> ...............
> 2012-09-15 14:21:08,667 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49944, bytes: 3216, op: HDFS_READ,
> cliID: DFSClient_36948932, offset: 139264, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_7326309033072036040_1074, duration: 753000 2012-09-15 14:21:11,671
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49947, bytes: 3096, op: HDFS_READ,
> cliID: DFSClient_36948932, offset: 139264, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_7326309033072036040_1074, duration: 372000 2012-09-15 14:21:11,672
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49948, bytes: 3216, op: HDFS_READ,
> cliID: DFSClient_36948932, offset: 139264, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_7326309033072036040_1074, duration: 353000 2012-09-15 14:21:11,673
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49949, bytes: 3216, op: HDFS_READ,
> cliID: DFSClient_36948932, offset: 139264, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_7326309033072036040_1074, duration: 355000 2012-09-15 14:21:14,677
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:50010, dest: /127.0.0.1:49951, bytes: 3216, op: HDFS_READ,
> cliID: DFSClient_36948932, offset: 139264, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_7326309033072036040_1074, duration: 368000 2012-09-15 14:21:17,628
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
> blk_5156635908233241378_1080 src: /127.0.0.1:49953 dest:
> /127.0.0.1:50010 2012-09-15 14:21:17,630 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
> /127.0.0.1:49953, dest: /127.0.0.1:50010, bytes: 28184, op:
> HDFS_WRITE, cliID: DFSClient_672168163, offset: 0, srvID:
> DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
> blk_5156635908233241378_1080, duration: 867000 2012-09-15 14:21:17,630
> INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
> 0 for block blk_5156635908233241378_1080 terminating 2012-09-15
> 14:21:21,128 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Scheduling block blk_-8793872286240925170_1064 file
> /Users/jasonhuang/hdfs/data/current/blk_-8793872286240925170 for
> deletion 2012-09-15 14:21:21,128 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_-6641115981657984283_1079 file
> /Users/jasonhuang/hdfs/data/current/blk_-6641115981657984283 for
> deletion 2012-09-15 14:21:21,128 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_-8793872286240925170_1064 at file
> /Users/jasonhuang/hdfs/data/current/blk_-8793872286240925170
> 2012-09-15 14:21:21,128 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_-6285781475981276067_1080 file
> /Users/jasonhuang/hdfs/data/current/blk_-6285781475981276067 for
> deletion 2012-09-15 14:21:21,128 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_-6282803597350612472_1068 file
> /Users/jasonhuang/hdfs/data/current/blk_-6282803597350612472 for
> deletion 2012-09-15 14:21:21,128 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_-6641115981657984283_1079 at file
> /Users/jasonhuang/hdfs/data/current/blk_-6641115981657984283
> 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_-1973500835155733464_1071 file
> /Users/jasonhuang/hdfs/data/current/blk_-1973500835155733464 for
> deletion 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_-6285781475981276067_1080 at file
> /Users/jasonhuang/hdfs/data/current/blk_-6285781475981276067
> 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_-201819473056990539_1072 file
> /Users/jasonhuang/hdfs/data/current/blk_-201819473056990539 for
> deletion 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_-6282803597350612472_1068 at file
> /Users/jasonhuang/hdfs/data/current/blk_-6282803597350612472
> 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_966543399440919118_1073 file
> /Users/jasonhuang/hdfs/data/current/blk_966543399440919118 for
> deletion 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_-1973500835155733464_1071 at file
> /Users/jasonhuang/hdfs/data/current/blk_-1973500835155733464
> 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
> blk_1230157759905402594_1069 file
> /Users/jasonhuang/hdfs/data/current/blk_1230157759905402594 for
> deletion 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_-201819473056990539_1072 at file
> /Users/jasonhuang/hdfs/data/current/blk_-201819473056990539 2012-09-15
> 14:21:21,129 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Scheduling block blk_3059764143082927316_1070 file
> /Users/jasonhuang/hdfs/data/current/blk_3059764143082927316 for
> deletion 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_966543399440919118_1073 at file
> /Users/jasonhuang/hdfs/data/current/blk_966543399440919118 2012-09-15
> 14:21:21,129 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Scheduling block blk_4471127410063335353_1066 file
> /Users/jasonhuang/hdfs/data/current/blk_4471127410063335353 for
> deletion 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_1230157759905402594_1069 at file
> /Users/jasonhuang/hdfs/data/current/blk_1230157759905402594 2012-09-15
> 14:21:21,129 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Scheduling block blk_5156635908233241378_1080 file
> /Users/jasonhuang/hdfs/data/current/blk_5156635908233241378 for
> deletion 2012-09-15 14:21:21,129 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_3059764143082927316_1070 at file
> /Users/jasonhuang/hdfs/data/current/blk_3059764143082927316 2012-09-15
> 14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Scheduling block blk_7335749996441800570_1065 file
> /Users/jasonhuang/hdfs/data/current/blk_7335749996441800570 for
> deletion 2012-09-15 14:21:21,130 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_4471127410063335353_1066 at file
> /Users/jasonhuang/hdfs/data/current/blk_4471127410063335353 2012-09-15
> 14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Scheduling block blk_7674314220695151815_1067 file
> /Users/jasonhuang/hdfs/data/current/blk_7674314220695151815 for
> deletion 2012-09-15 14:21:21,130 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
> blk_5156635908233241378_1080 at file
> /Users/jasonhuang/hdfs/data/current/blk_5156635908233241378 2012-09-15
> 14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Deleted block blk_7335749996441800570_1065 at file
> /Users/jasonhuang/hdfs/data/current/blk_7335749996441800570 2012-09-15
> 14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Deleted block blk_7674314220695151815_1067 at file
> /Users/jasonhuang/hdfs/data/current/blk_7674314220695151815 2012-09-15
> 14:29:48,016 INFO
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
> succeeded for blk_5700986404331589806_1038
>
> Not sure where I should go next. Hope to get some help.
>
> I didn't  weak any checksum size related configs in my config files -
> actually I am not even aware of the ability to do in config files.
>
> thanks!
>
> Jason
>
>
> On Fri, Sep 14, 2012 at 10:05 PM, Harsh J <harsh@cloudera.com> wrote:
>> Hi Jason,
>>
>> Does the DN log have something in it that corresponds to these errors?
>> Is there also some stacktrace/further text after the line you've
>> pasted until? Can we have it?
>>
>> Also, did you tweak any checksum size related configs in your config files?
>>
>> On Sat, Sep 15, 2012 at 3:20 AM, Jason Huang <jason.huang@icare.com> wrote:
>>> Hello,
>>>
>>> Looking for some help in setting up hadoop 1.0.3 in Pseudo distributed mode...
>>>
>>> I was able to install hadoop, config the .xml files and start all nodes:
>>> $ JPS
>>> 6645 Jps
>>> 6030 SecondaryNameNode
>>> 6185 TaskTracker
>>> 5851 NameNode
>>> 6095 JobTracker
>>> 5939 DataNode
>>>
>>> However, when I tried to play around with a couple of Map-reduce jobs
>>> with provided example jar files I got the following errors:
>>>
>>> (1) $ bin/hadoop jar hadoop-examples-1.0.3.jar pi 10 100
>>> Number of Maps  = 10
>>> Samples per Map = 100
>>> Wrote input for Map #0
>>> Wrote input for Map #1
>>> Wrote input for Map #2
>>> Wrote input for Map #3
>>> Wrote input for Map #4
>>> Wrote input for Map #5
>>> Wrote input for Map #6
>>> Wrote input for Map #7
>>> Wrote input for Map #8
>>> Wrote input for Map #9
>>> Starting Job
>>> 12/09/14 17:39:06 INFO mapred.FileInputFormat: Total input paths to process :
10
>>> 12/09/14 17:39:06 INFO mapred.JobClient: Running job: job_201209141701_0003
>>> 12/09/14 17:39:07 INFO mapred.JobClient:  map 0% reduce 0%
>>> 12/09/14 17:39:16 INFO mapred.JobClient: Task Id :
>>> attempt_201209141701_0003_m_000011_0, Status : FAILED
>>> Error initializing attempt_201209141701_0003_m_000011_0:
>>> java.io.IOException: BlockReader: error in packet header(chunkOffset :
>>> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>>>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>>>
>>> (2) $ ./bin/hadoop jar hadoop-examples-1.0.3.jar wordcount
>>> /user/jasonhuang/input /user/jasonhuang/output
>>> 12/09/14 17:37:51 INFO input.FileInputFormat: Total input paths to process :
1
>>> 12/09/14 17:37:51 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop library for your platform... using builtin-java classes
>>> where applicable
>>> 12/09/14 17:37:51 WARN snappy.LoadSnappy: Snappy native library not loaded
>>> 12/09/14 17:37:57 INFO mapred.JobClient: Cleaning up the staging area
>>> hdfs://localhost:9000/tmp/hadoop-jasonhuang/mapred/staging/jasonhuang/.staging/job_201209141701_0002
>>> 12/09/14 17:37:57 ERROR security.UserGroupInformation:
>>> PriviledgedActionException as:jasonhuang
>>> cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>>> java.io.IOException: BlockReader: error in packet header(chunkOffset :
>>> 19968, dataLen : 1835351087, seqno : 7023413562532324724 (last: 0))
>>>
>>> Does anyone have idea on why the error occurs and how I can fix them?
>>>
>>> thanks!
>>>
>>> Jason
>>
>>
>>
>> --
>> Harsh J

Mime
View raw message