hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Huang <jason.hu...@icare.com>
Subject Re: HDFS Error - BlockReader: error in packet header
Date Sat, 15 Sep 2012 18:37:59 GMT
Thanks Harsh.

I've tried the following again:
$ ./bin/hadoop jar hadoop-examples-1.0.3.jar pi 10 100

And I got the same error (sorry for having to paste this longggg log):
Number of Maps  = 10
Samples per Map = 100
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
12/09/15 14:20:02 INFO mapred.FileInputFormat: Total input paths to process : 10
12/09/15 14:20:02 INFO mapred.JobClient: Running job: job_201209151409_0001
12/09/15 14:20:03 INFO mapred.JobClient:  map 0% reduce 0%
12/09/15 14:20:14 INFO mapred.JobClient: Task Id :
attempt_201209151409_0001_m_000011_0, Status : FAILED
Error initializing attempt_201209151409_0001_m_000011_0:
java.io.IOException: BlockReader: error in packet header(chunkOffset :
142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
	at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
	at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
	at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
	at java.lang.Thread.run(Thread.java:636)

12/09/15 14:20:14 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_0&filter=stdout
12/09/15 14:20:14 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_0&filter=stderr
12/09/15 14:20:23 INFO mapred.JobClient: Task Id :
attempt_201209151409_0001_m_000011_1, Status : FAILED
Error initializing attempt_201209151409_0001_m_000011_1:
java.io.IOException: BlockReader: error in packet header(chunkOffset :
142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
	at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
	at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
	at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
	at java.lang.Thread.run(Thread.java:636)

12/09/15 14:20:23 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_1&filter=stdout
12/09/15 14:20:23 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_1&filter=stderr
12/09/15 14:20:32 INFO mapred.JobClient: Task Id :
attempt_201209151409_0001_m_000011_2, Status : FAILED
Error initializing attempt_201209151409_0001_m_000011_2:
java.io.IOException: BlockReader: error in packet header(chunkOffset :
142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
	at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
	at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
	at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
	at java.lang.Thread.run(Thread.java:636)

12/09/15 14:20:32 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_2&filter=stdout
12/09/15 14:20:32 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000011_2&filter=stderr
12/09/15 14:20:50 INFO mapred.JobClient: Task Id :
attempt_201209151409_0001_m_000010_0, Status : FAILED
Error initializing attempt_201209151409_0001_m_000010_0:
java.io.IOException: BlockReader: error in packet header(chunkOffset :
142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
	at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
	at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
	at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
	at java.lang.Thread.run(Thread.java:636)

12/09/15 14:20:50 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_0&filter=stdout
12/09/15 14:20:50 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_0&filter=stderr
12/09/15 14:20:59 INFO mapred.JobClient: Task Id :
attempt_201209151409_0001_m_000010_1, Status : FAILED
Error initializing attempt_201209151409_0001_m_000010_1:
java.io.IOException: BlockReader: error in packet header(chunkOffset :
142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
	at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
	at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
	at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
	at java.lang.Thread.run(Thread.java:636)

12/09/15 14:20:59 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_1&filter=stdout
12/09/15 14:20:59 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_1&filter=stderr
12/09/15 14:21:08 INFO mapred.JobClient: Task Id :
attempt_201209151409_0001_m_000010_2, Status : FAILED
Error initializing attempt_201209151409_0001_m_000010_2:
java.io.IOException: BlockReader: error in packet header(chunkOffset :
142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
	at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
	at org.apache.hadoop.hdfs.DFSClient$BlockReader.read(DFSClient.java:1460)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.readBuffer(DFSClient.java:2175)
	at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2227)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:74)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:230)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1222)
	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1203)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobJarFile(JobLocalizer.java:273)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:377)
	at org.apache.hadoop.mapred.JobLocalizer.localizeJobFiles(JobLocalizer.java:367)
	at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:202)
	at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1228)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:416)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1203)
	at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1118)
	at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2430)
	at java.lang.Thread.run(Thread.java:636)

12/09/15 14:21:08 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_2&filter=stdout
12/09/15 14:21:08 WARN mapred.JobClient: Error reading task
outputhttp://192.168.1.124:50060/tasklog?plaintext=true&attemptid=attempt_201209151409_0001_m_000010_2&filter=stderr
12/09/15 14:21:17 INFO mapred.JobClient: Job complete: job_201209151409_0001
12/09/15 14:21:17 INFO mapred.JobClient: Counters: 4
12/09/15 14:21:17 INFO mapred.JobClient:   Job Counters
12/09/15 14:21:17 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=0
12/09/15 14:21:17 INFO mapred.JobClient:     Total time spent by all
reduces waiting after reserving slots (ms)=0
12/09/15 14:21:17 INFO mapred.JobClient:     Total time spent by all
maps waiting after reserving slots (ms)=0
12/09/15 14:21:17 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
12/09/15 14:21:17 INFO mapred.JobClient: Job Failed: JobCleanup Task
Failure, Task: task_201209151409_0001_m_000010
java.io.IOException: Job failed!
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
	at org.apache.hadoop.examples.PiEstimator.estimate(PiEstimator.java:297)
	at org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:342)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:351)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)


Here's the a 'brief' version of the data node log (but still very long...):
2012-09-15 14:20:02,640 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_39584416619615086_1077 src: /127.0.0.1:49829 dest:
/127.0.0.1:50010 2012-09-15 14:20:02,642 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:49829, dest: /127.0.0.1:50010, bytes: 20494, op:
HDFS_WRITE, cliID: DFSClient_1015299679, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_39584416619615086_1077, duration: 565000 2012-09-15 14:20:02,642
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
0 for block blk_39584416619615086_1077 terminating 2012-09-15
14:20:02,663 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49831, bytes: 20658, op: HDFS_READ,
cliID: DFSClient_672168163, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_39584416619615086_1077, duration: 270000 2012-09-15 14:20:02,665
INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49832, bytes: 18, op: HDFS_READ,
cliID: DFSClient_672168163, offset: 20480, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_39584416619615086_1077, duration: 146000 2012-09-15 14:20:02,757
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-6641115981657984283_1079 src: /127.0.0.1:49833 dest:
/127.0.0.1:50010 2012-09-15 14:20:02,761 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:49833, dest: /127.0.0.1:50010, bytes: 20563, op:
HDFS_WRITE, cliID: DFSClient_672168163, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_-6641115981657984283_1079, duration: 2189000 2012-09-15
14:20:02,761 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder 0 for block blk_-6641115981657984283_1079 terminating
2012-09-15 14:20:02,776 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-6285781475981276067_1080 src: /127.0.0.1:49835 dest:
/127.0.0.1:50010 2012-09-15 14:20:02,777 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:49835, dest: /127.0.0.1:50010, bytes: 106, op: HDFS_WRITE,
cliID: DFSClient_1436299339, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_-6285781475981276067_1080, duration: 321000 2012-09-15
14:20:02,777 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder 0 for block blk_-6285781475981276067_1080 terminating
2012-09-15 14:20:02,781 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49836, bytes: 201, op: HDFS_READ,
cliID: DFSClient_672168163, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_-6638660277808139598_1076, duration: 152000 2012-09-15
14:20:05,555 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49838, bytes: 110, op: HDFS_READ,
cliID: DFSClient_1214970016, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_-6285781475981276067_1080, duration: 158000 2012-09-15
14:20:05,563 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49840, bytes: 20658, op: HDFS_READ,
cliID: DFSClient_1762809953, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_39584416619615086_1077, duration: 302000  ................
...............
2012-09-15 14:21:08,667 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49944, bytes: 3216, op: HDFS_READ,
cliID: DFSClient_36948932, offset: 139264, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_7326309033072036040_1074, duration: 753000 2012-09-15 14:21:11,671
INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49947, bytes: 3096, op: HDFS_READ,
cliID: DFSClient_36948932, offset: 139264, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_7326309033072036040_1074, duration: 372000 2012-09-15 14:21:11,672
INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49948, bytes: 3216, op: HDFS_READ,
cliID: DFSClient_36948932, offset: 139264, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_7326309033072036040_1074, duration: 353000 2012-09-15 14:21:11,673
INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49949, bytes: 3216, op: HDFS_READ,
cliID: DFSClient_36948932, offset: 139264, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_7326309033072036040_1074, duration: 355000 2012-09-15 14:21:14,677
INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:50010, dest: /127.0.0.1:49951, bytes: 3216, op: HDFS_READ,
cliID: DFSClient_36948932, offset: 139264, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_7326309033072036040_1074, duration: 368000 2012-09-15 14:21:17,628
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_5156635908233241378_1080 src: /127.0.0.1:49953 dest:
/127.0.0.1:50010 2012-09-15 14:21:17,630 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/127.0.0.1:49953, dest: /127.0.0.1:50010, bytes: 28184, op:
HDFS_WRITE, cliID: DFSClient_672168163, offset: 0, srvID:
DS-1101353210-192.168.10.23-50010-1347651592008, blockid:
blk_5156635908233241378_1080, duration: 867000 2012-09-15 14:21:17,630
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder
0 for block blk_5156635908233241378_1080 terminating 2012-09-15
14:21:21,128 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling block blk_-8793872286240925170_1064 file
/Users/jasonhuang/hdfs/data/current/blk_-8793872286240925170 for
deletion 2012-09-15 14:21:21,128 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
blk_-6641115981657984283_1079 file
/Users/jasonhuang/hdfs/data/current/blk_-6641115981657984283 for
deletion 2012-09-15 14:21:21,128 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_-8793872286240925170_1064 at file
/Users/jasonhuang/hdfs/data/current/blk_-8793872286240925170
2012-09-15 14:21:21,128 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
blk_-6285781475981276067_1080 file
/Users/jasonhuang/hdfs/data/current/blk_-6285781475981276067 for
deletion 2012-09-15 14:21:21,128 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
blk_-6282803597350612472_1068 file
/Users/jasonhuang/hdfs/data/current/blk_-6282803597350612472 for
deletion 2012-09-15 14:21:21,128 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_-6641115981657984283_1079 at file
/Users/jasonhuang/hdfs/data/current/blk_-6641115981657984283
2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
blk_-1973500835155733464_1071 file
/Users/jasonhuang/hdfs/data/current/blk_-1973500835155733464 for
deletion 2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_-6285781475981276067_1080 at file
/Users/jasonhuang/hdfs/data/current/blk_-6285781475981276067
2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
blk_-201819473056990539_1072 file
/Users/jasonhuang/hdfs/data/current/blk_-201819473056990539 for
deletion 2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_-6282803597350612472_1068 at file
/Users/jasonhuang/hdfs/data/current/blk_-6282803597350612472
2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
blk_966543399440919118_1073 file
/Users/jasonhuang/hdfs/data/current/blk_966543399440919118 for
deletion 2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_-1973500835155733464_1071 at file
/Users/jasonhuang/hdfs/data/current/blk_-1973500835155733464
2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block
blk_1230157759905402594_1069 file
/Users/jasonhuang/hdfs/data/current/blk_1230157759905402594 for
deletion 2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_-201819473056990539_1072 at file
/Users/jasonhuang/hdfs/data/current/blk_-201819473056990539 2012-09-15
14:21:21,129 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling block blk_3059764143082927316_1070 file
/Users/jasonhuang/hdfs/data/current/blk_3059764143082927316 for
deletion 2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_966543399440919118_1073 at file
/Users/jasonhuang/hdfs/data/current/blk_966543399440919118 2012-09-15
14:21:21,129 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling block blk_4471127410063335353_1066 file
/Users/jasonhuang/hdfs/data/current/blk_4471127410063335353 for
deletion 2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_1230157759905402594_1069 at file
/Users/jasonhuang/hdfs/data/current/blk_1230157759905402594 2012-09-15
14:21:21,129 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling block blk_5156635908233241378_1080 file
/Users/jasonhuang/hdfs/data/current/blk_5156635908233241378 for
deletion 2012-09-15 14:21:21,129 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_3059764143082927316_1070 at file
/Users/jasonhuang/hdfs/data/current/blk_3059764143082927316 2012-09-15
14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling block blk_7335749996441800570_1065 file
/Users/jasonhuang/hdfs/data/current/blk_7335749996441800570 for
deletion 2012-09-15 14:21:21,130 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_4471127410063335353_1066 at file
/Users/jasonhuang/hdfs/data/current/blk_4471127410063335353 2012-09-15
14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Scheduling block blk_7674314220695151815_1067 file
/Users/jasonhuang/hdfs/data/current/blk_7674314220695151815 for
deletion 2012-09-15 14:21:21,130 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block
blk_5156635908233241378_1080 at file
/Users/jasonhuang/hdfs/data/current/blk_5156635908233241378 2012-09-15
14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Deleted block blk_7335749996441800570_1065 at file
/Users/jasonhuang/hdfs/data/current/blk_7335749996441800570 2012-09-15
14:21:21,130 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Deleted block blk_7674314220695151815_1067 at file
/Users/jasonhuang/hdfs/data/current/blk_7674314220695151815 2012-09-15
14:29:48,016 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_5700986404331589806_1038

Not sure where I should go next. Hope to get some help.

I didn't  weak any checksum size related configs in my config files -
actually I am not even aware of the ability to do in config files.

thanks!

Jason


On Fri, Sep 14, 2012 at 10:05 PM, Harsh J <harsh@cloudera.com> wrote:
> Hi Jason,
>
> Does the DN log have something in it that corresponds to these errors?
> Is there also some stacktrace/further text after the line you've
> pasted until? Can we have it?
>
> Also, did you tweak any checksum size related configs in your config files?
>
> On Sat, Sep 15, 2012 at 3:20 AM, Jason Huang <jason.huang@icare.com> wrote:
>> Hello,
>>
>> Looking for some help in setting up hadoop 1.0.3 in Pseudo distributed mode...
>>
>> I was able to install hadoop, config the .xml files and start all nodes:
>> $ JPS
>> 6645 Jps
>> 6030 SecondaryNameNode
>> 6185 TaskTracker
>> 5851 NameNode
>> 6095 JobTracker
>> 5939 DataNode
>>
>> However, when I tried to play around with a couple of Map-reduce jobs
>> with provided example jar files I got the following errors:
>>
>> (1) $ bin/hadoop jar hadoop-examples-1.0.3.jar pi 10 100
>> Number of Maps  = 10
>> Samples per Map = 100
>> Wrote input for Map #0
>> Wrote input for Map #1
>> Wrote input for Map #2
>> Wrote input for Map #3
>> Wrote input for Map #4
>> Wrote input for Map #5
>> Wrote input for Map #6
>> Wrote input for Map #7
>> Wrote input for Map #8
>> Wrote input for Map #9
>> Starting Job
>> 12/09/14 17:39:06 INFO mapred.FileInputFormat: Total input paths to process : 10
>> 12/09/14 17:39:06 INFO mapred.JobClient: Running job: job_201209141701_0003
>> 12/09/14 17:39:07 INFO mapred.JobClient:  map 0% reduce 0%
>> 12/09/14 17:39:16 INFO mapred.JobClient: Task Id :
>> attempt_201209141701_0003_m_000011_0, Status : FAILED
>> Error initializing attempt_201209141701_0003_m_000011_0:
>> java.io.IOException: BlockReader: error in packet header(chunkOffset :
>> 142336, dataLen : 3538944, seqno : 3350829872548206857 (last: 0))
>>         at org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1580)
>>
>> (2) $ ./bin/hadoop jar hadoop-examples-1.0.3.jar wordcount
>> /user/jasonhuang/input /user/jasonhuang/output
>> 12/09/14 17:37:51 INFO input.FileInputFormat: Total input paths to process : 1
>> 12/09/14 17:37:51 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes
>> where applicable
>> 12/09/14 17:37:51 WARN snappy.LoadSnappy: Snappy native library not loaded
>> 12/09/14 17:37:57 INFO mapred.JobClient: Cleaning up the staging area
>> hdfs://localhost:9000/tmp/hadoop-jasonhuang/mapred/staging/jasonhuang/.staging/job_201209141701_0002
>> 12/09/14 17:37:57 ERROR security.UserGroupInformation:
>> PriviledgedActionException as:jasonhuang
>> cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>> java.io.IOException: BlockReader: error in packet header(chunkOffset :
>> 19968, dataLen : 1835351087, seqno : 7023413562532324724 (last: 0))
>>
>> Does anyone have idea on why the error occurs and how I can fix them?
>>
>> thanks!
>>
>> Jason
>
>
>
> --
> Harsh J

Mime
View raw message