Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C2391881A for ; Tue, 15 Sep 2015 15:21:00 +0000 (UTC) Received: (qmail 5544 invoked by uid 500); 15 Sep 2015 15:20:39 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 5447 invoked by uid 500); 15 Sep 2015 15:20:39 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 5437 invoked by uid 99); 15 Sep 2015 15:20:39 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Sep 2015 15:20:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 2AB4B180333 for ; Tue, 15 Sep 2015 15:20:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.526 X-Spam-Level: X-Spam-Status: No, score=-4.526 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_UNSUB1=0.1, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.006, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=cisco.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id nOndqX5bN8L8 for ; Tue, 15 Sep 2015 15:20:29 +0000 (UTC) Received: from alln-iport-1.cisco.com (alln-iport-1.cisco.com [173.37.142.88]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 61349206E7 for ; Tue, 15 Sep 2015 15:20:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=19524; q=dns/txt; s=iport; t=1442330429; x=1443540029; h=from:to:subject:date:message-id:mime-version; bh=CmZWGTejyDhLZpBTu2oTacK6qp7w4ndF7Bxzz9SCg3I=; b=Ga6LQwZCorLajcBozjI5F/BRZNhTcGUZWck02my2JzIP6DHYIpoDRR6c zUJLVBTYd+xYUctKNgfx9r/NgV58dhRqmUaSEtywHMTev4PwT/x0WNHPY gHx4TGgZby9y3fUD7EoOOnaAehSbMqBaA/cYhcjTpeDkFgERoEt2UmoHs 4=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0D6AQDdNvhV/5ldJa1aA4JWTVRpBr1HAQ2BbwELgkGDNoFAOBQBAQEBAQEBfwuEKmkBBB0BDzg5FBMEE4guDaZTkTWDaY5pAQEBAQYBAQEBAQEcgTaBGYQkiScRASsWKYIcTYExBYcyhUY5iCsBhQ+Hc4FNRoNtgyGEAY1jAR8BAUKEAXEBiGMHFyOBBQEBAQ X-IronPort-AV: E=Sophos;i="5.17,536,1437436800"; d="scan'208,217";a="188413943" Received: from rcdn-core-2.cisco.com ([173.37.93.153]) by alln-iport-1.cisco.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 15 Sep 2015 15:20:21 +0000 Received: from XCH-ALN-003.cisco.com (xch-aln-003.cisco.com [173.36.7.13]) by rcdn-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id t8FFKKu0008237 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL) for ; Tue, 15 Sep 2015 15:20:20 GMT Received: from xch-aln-001.cisco.com (173.36.7.11) by XCH-ALN-003.cisco.com (173.36.7.13) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Tue, 15 Sep 2015 10:20:19 -0500 Received: from xch-aln-001.cisco.com ([173.36.7.11]) by XCH-ALN-001.cisco.com ([173.36.7.11]) with mapi id 15.00.1104.000; Tue, 15 Sep 2015 10:20:19 -0500 From: "Grant Overby (groverby)" To: "user@hadoop.apache.org" Subject: BlockMissingException, but file seems fine. Thread-Topic: BlockMissingException, but file seems fine. Thread-Index: AQHQ78oI82fOeIedvUu97/Y7G5D4Dw== Date: Tue, 15 Sep 2015 15:20:19 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.150.181.66] Content-Type: multipart/alternative; boundary="_000_D21DAF901E02Dgroverbyciscocom_" MIME-Version: 1.0 --_000_D21DAF901E02Dgroverbyciscocom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I'm getting the following exception when using Drill. Asked on Drill mailin= g list a few days back; no one had any ideas. As near as I can tell, the block is there and retrievable. Any thoughts on = what could be causing this exception. This is the stack trace: Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain b= lock: BP-1605794487-10.0.1.3-1435700184285:blk_1073829560_90288 file=3D/war= ehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a4= 9ba-3697-47ff-ab2c-614281346ff6.parquet at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStr= eam.java:880) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream= .java:560) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.seekToNewSource(DFSInputSt= ream.java:1443) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.= java:766) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputS= tream.java:796) ~[hadoop-hdfs-2.4.1.jar:na] at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:8= 37) ~[hadoop-hdfs-2.4.1.jar:na] at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.= 0_79] at org.apache.drill.exec.store.dfs.DrillFSDataInputStream$WrappedIn= putStream.read(DrillFSDataInputStream.java:211) ~[drill-java-exec-1.1.0.jar= :1.1.0] at java.io.DataInputStream.read(DataInputStream.java:149) ~[na:1.7.= 0_79] at parquet.org.apache.thrift.transport.TIOStreamTransport.read(TIOS= treamTransport.java:127) ~[parquet-format-2.1.1-drill-r1.jar:na] ... 40 common frames omitted This is a two node cluster, both nodes can retrieve the file. root@twig03:/opt/drill/log# hdfs dfs -get /warehouse2/completed/events/conn= ection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346f= f6.parquet /tmp/. root@twig03:/opt/drill/log# ls /tmp/*.parquet /tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet root@twig04:~# hdfs dfs -get /warehouse2/completed/events/connection_events= /1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet /t= mp/. root@twig04:~# ls /tmp/*.parquet /tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet hdfs reports no missing blocks: root@twig04:~# hdfs dfsadmin -report Configured Capacity: 7856899358720 (7.15 TB) Present Capacity: 7856899358720 (7.15 TB) DFS Remaining: 4310190084096 (3.92 TB) DFS Used: 3546709274624 (3.23 TB) DFS Used%: 45.14% Under replicated blocks: 23854 Blocks with corrupt replicas: 0 Missing blocks: 0 ------------------------------------------------- Live datanodes (2): Name: 10.0.1.4:50010 (twig04.twigs) Hostname: twig04.twigs Decommission Status : Normal Configured Capacity: 3928449679360 (3.57 TB) DFS Used: 1773355933696 (1.61 TB) Non DFS Used: 0 (0 B) DFS Remaining: 2155093745664 (1.96 TB) DFS Used%: 45.14% DFS Remaining%: 54.86% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 3 Last contact: Tue Sep 15 11:18:00 EDT 2015 Name: 10.0.1.3:50010 (twig03.twigs) Hostname: twig03.twigs Decommission Status : Normal Configured Capacity: 3928449679360 (3.57 TB) DFS Used: 1773353340928 (1.61 TB) Non DFS Used: 0 (0 B) DFS Remaining: 2155096338432 (1.96 TB) DFS Used%: 45.14% DFS Remaining%: 54.86% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 2 Last contact: Tue Sep 15 11:18:02 EDT 2015 [http://www.cisco.com/web/europe/images/email/signature/est2014/logo_06.png= ?ct=3D1398192119726] Grant Overby Software Engineer Cisco.com groverby@cisco.com Mobile: 865 724 4910 [http://www.cisco.com/assets/swa/img/thinkbeforeyouprint.gif] Think before = you print. This email may contain confidential and privileged material for the sole us= e of the intended recipient. Any review, use, distribution or disclosure by= others is strictly prohibited. If you are not the intended recipient (or a= uthorized to receive for the recipient), please contact the sender by reply= email and delete all copies of this message. Please click here for Company Registration Information. --_000_D21DAF901E02Dgroverbyciscocom_ Content-Type: text/html; charset="iso-8859-1" Content-ID: <718700DED4869D43AD44B096B860DFCA@emea.cisco.com> Content-Transfer-Encoding: quoted-printable
I’m getting the following exception when using Drill. Asked on D= rill mailing list a few days back; no one had any ideas.

As near as I can tell, the block is there and retrievable. Any thought= s on what could be causing this exception. 

This is the stack trace:

Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain b= lock: BP-1605794487-10.0.1.3-1435700184285:blk_1073829560_90288 file=3D/war= ehouse2/completed/events/connection_events/1441747800/1441747805625-1-fe3a4= 9ba-3697-47ff-ab2c-614281346ff6.parquet

        at org.apache.hadoop.hdfs.DFSInputStream.choose= DataNode(DFSInputStream.java:880) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.blockS= eekTo(DFSInputStream.java:560) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.seekTo= NewSource(DFSInputStream.java:1443) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.readBu= ffer(DFSInputStream.java:766) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.readWi= thStrategy(DFSInputStream.java:796) ~[hadoop-hdfs-2.4.1.jar:na]

        at org.apache.hadoop.hdfs.DFSInputStream.read(D= FSInputStream.java:837) ~[hadoop-hdfs-2.4.1.jar:na]

        at java.io.DataInputStream.read(DataInputStream= .java:149) ~[na:1.7.0_79]

        at org.apache.drill.exec.store.dfs.DrillFSDataI= nputStream$WrappedInputStream.read(DrillFSDataInputStream.java:211) ~[drill= -java-exec-1.1.0.jar:1.1.0]

        at java.io.DataInputStream.read(DataInputStream= .java:149) ~[na:1.7.0_79]

        at parquet.org.apache.thrift.transport.TIOStrea= mTransport.read(TIOStreamTransport.java:127) ~[parquet-format-2.1.1-drill-r= 1.jar:na]

        ... 40 common frames omitted




This is a two node cluster, both nodes can retrieve the file.

root@twig03:/opt/drill/log# hdfs dfs -get /warehouse2/completed/events/conn= ection_events/1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346f= f6.parquet /tmp/.

root@twig03:/opt/drill/log# ls /tmp/*.parquet

/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet


root@twig04:~# hdfs dfs -get /warehouse2/completed/events/connection_events= /1441747800/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet /t= mp/.

root@twig04:~# ls /tmp/*.parquet

/tmp/1441747805625-1-fe3a49ba-3697-47ff-ab2c-614281346ff6.parquet




hdfs reports no missing blocks:

root@twig04:~# hdfs dfsadmin -report

Configured Capacity: 7856899358720 (7.15 TB)

Present Capacity: 7856899358720 (7.15 TB)

DFS Remaining: 4310190084096 (3.92 TB)

DFS Used: 3546709274624 (3.23 TB)

DFS Used%: 45.14%

Under replicated blocks: 23854

Blocks with corrupt replicas: 0

Missing blocks: 0


-------------------------------------------------

Live datanodes (2):


Name: 10.0.1.4:50010 (twig04.twigs)

Hostname: twig04.twigs

Decommission Status : Normal

Configured Capacity: 3928449679360 (3.57 TB)

DFS Used: 1773355933696 (1.61 TB)

Non DFS Used: 0 (0 B)

DFS Remaining: 2155093745664 (1.96 TB)

DFS Used%: 45.14%

DFS Remaining%: 54.86%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 3

Last contact: Tue Sep 15 11:18:00 EDT 2015



Name: 10.0.1.3:50010 (twig03.twigs)

Hostname: twig03.twigs

Decommission Status : Normal

Configured Capacity: 3928449679360 (3.57 TB)

DFS Used: 1773353340928 (1.61 TB)

Non DFS Used: 0 (0 B)

DFS Remaining: 2155096338432 (1.96 TB)

DFS Used%: 45.14%

DFS Remaining%: 54.86%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 2

Last contact: Tue Sep 15 11:18:02 EDT 2015









Grant Overby
Software Engineer
Cisco.com
groverby@cisco.com
Mobile: 865 724 4910



 Think before you print.

This email may contain confidential and privileged material for the sole= use of the intended recipient. Any review, use, distribution or disclosure= by others is strictly prohibited. If you are not the intended recipient (o= r authorized to receive for the recipient), please contact the sender by reply email and delete all copies= of this message.

Please click here for Company Registration Information.


--_000_D21DAF901E02Dgroverbyciscocom_--