hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grant Overby (groverby)" <grove...@cisco.com>
Subject BlockMissingException, but file seems fine.
Date Tue, 15 Sep 2015 15:20:19 GMT
I'm getting the following exception when using Drill. Asked on Drill mailing list a few days
back; no one had any ideas.

As near as I can tell, the block is there and retrievable. Any thoughts on what could be causing
this exception.

This is the stack trace:

Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1605794487-10.0.1.3-1435700184285:blk_1073829560_90288
file=/warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet

        at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:880) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:560) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.seekToNewSource(DFSInputStream.java:1443)
~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:766) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:796)
~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:837) ~[hadoop-hdfs-2.4.1.jar:na]

        at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.0_79]

        at org.apache.drill.exec.store.dfs.DrillFSDataInputStream$WrappedInputStream.read(DrillFSDataInputStream.java:211)
~[drill-java-exec-1.1.0.jar:1.1.0]

        at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.0_79]

        at parquet.org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
~[parquet-format-2.1.1-drill-r1.jar:na]

        ... 40 common frames omitted



This is a two node cluster, both nodes can retrieve the file.

root@twig03:/opt/drill/log# hdfs dfs -get /warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
/tmp/.

root@twig03:/opt/drill/log# ls /tmp/*.parquet

/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet


root@twig04:~# hdfs dfs -get /warehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet
/tmp/.

root@twig04:~# ls /tmp/*.parquet

/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet



hdfs reports no missing blocks:

root@twig04:~# hdfs dfsadmin -report

Configured Capacity: 7856899358720 (7.15 TB)

Present Capacity: 7856899358720 (7.15 TB)

DFS Remaining: 4310190084096 (3.92 TB)

DFS Used: 3546709274624 (3.23 TB)

DFS Used%: 45.14%

Under replicated blocks: 23854

Blocks with corrupt replicas: 0

Missing blocks: 0


-------------------------------------------------

Live datanodes (2):


Name: 10.0.1.4:50010 (twig04.twigs)

Hostname: twig04.twigs

Decommission Status : Normal

Configured Capacity: 3928449679360 (3.57 TB)

DFS Used: 1773355933696 (1.61 TB)

Non DFS Used: 0 (0 B)

DFS Remaining: 2155093745664 (1.96 TB)

DFS Used%: 45.14%

DFS Remaining%: 54.86%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 3

Last contact: Tue Sep 15 11:18:00 EDT 2015



Name: 10.0.1.3:50010 (twig03.twigs)

Hostname: twig03.twigs

Decommission Status : Normal

Configured Capacity: 3928449679360 (3.57 TB)

DFS Used: 1773353340928 (1.61 TB)

Non DFS Used: 0 (0 B)

DFS Remaining: 2155096338432 (1.96 TB)

DFS Used%: 45.14%

DFS Remaining%: 54.86%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 2

Last contact: Tue Sep 15 11:18:02 EDT 2015








[http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png?ct=1398192119726]

Grant Overby
Software Engineer
Cisco.com<http://www.cisco.com/>
groverby@cisco.com<mailto:groverby@cisco.com>
Mobile: 865 724 4910






[http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before you print.

This email may contain confidential and privileged material for the sole use of the intended
recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If
you are not the intended recipient (or authorized to receive for the recipient), please contact
the sender by reply email and delete all copies of this message.

Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html>
for Company Registration Information.





Mime
View raw message