Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3496310F54 for ; Wed, 11 Dec 2013 09:17:19 +0000 (UTC) Received: (qmail 61188 invoked by uid 500); 11 Dec 2013 09:17:10 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 61029 invoked by uid 500); 11 Dec 2013 09:17:10 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 60984 invoked by uid 99); 11 Dec 2013 09:17:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 09:17:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of justlooks@gmail.com designates 209.85.128.51 as permitted sender) Received: from [209.85.128.51] (HELO mail-qe0-f51.google.com) (209.85.128.51) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Dec 2013 09:17:05 +0000 Received: by mail-qe0-f51.google.com with SMTP id 1so5171397qee.10 for ; Wed, 11 Dec 2013 01:16:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=P+lHw4bjEmrt+lF0c75jogFi4+B2yinG0YThoGIHx9c=; b=T97kubdcQQOKkZl/O9TYFDJqVq+2gP8wbVSW31MKeGIWIKEalxWMGTU+A8rVFa0gjB hKFTgsEZcl0pS7r1ZPgE9QEbRHzaA4XdBE4xJvyjoyqrZHmt8VB0t0PtaAJIFoutNn/V CAEo1fWJPC1lnsMWl8/CAJJyARHq56sAWXGDJrtaXc06j+0Xtkfe9C1P3iIHcmWZIw8P h22tg9QXOVHOleZj/jeRoVpDxsaHas09MyRLJSMjC7lWnqTIIjpUcfoNnmxd+c8O5LZU 6jobp+2IRQWKy9sT6Wlu1/5avHisWJmqdYtMj4ykdohZ+WWWtCiMsoxALeT9lEiM/xBE oyFQ== MIME-Version: 1.0 X-Received: by 10.224.0.68 with SMTP id 4mr592526qaa.17.1386753404987; Wed, 11 Dec 2013 01:16:44 -0800 (PST) Received: by 10.140.20.37 with HTTP; Wed, 11 Dec 2013 01:16:44 -0800 (PST) In-Reply-To: References: Date: Wed, 11 Dec 2013 17:16:44 +0800 Message-ID: Subject: Re: issue about corrupt block test From: ch huang To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7bf0ca7c186cfc04ed3eb0bb X-Virus-Checked: Checked by ClamAV on apache.org --047d7bf0ca7c186cfc04ed3eb0bb Content-Type: text/plain; charset=ISO-8859-1 you are right ,but i only find meta file why no block data file? # find /data -name 'blk_-5451264646515882190_106793*' /data/dataspace/3/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/subdir39/blk_-5451264646515882190_106793.meta # ls /data/dataspace/3/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/subdir39/ blk_3810334848964580951 blk_4621466474283145207_106809.meta blk_-5451264646515882190 blk_580162309124277323_106788.meta blk_3810334848964580951_106801.meta blk_516060569193828059 blk_-5451264646515882190_106793.meta blk_4621466474283145207 blk_516060569193828059_106796.meta blk_580162309124277323 On Wed, Dec 11, 2013 at 3:16 PM, Harsh J wrote: > Block files are not stored in a flat directory (to avoid FS limits of > max files under a dir). Instead of looking for them right under > finalized, issue a "find" query with the pattern instead and you > should be able to spot it. > > On Wed, Dec 11, 2013 at 9:10 AM, ch huang wrote: > > hi,maillist: > > i try to corrupt a block of a file in my benchmark > environment, > > as the following command i find blk_2504407693800874616_106252 ,it's > replica > > on 192.168.10.224 is my target ,but i find all the datadir in > 192.168.10.224 > > ,can not fine the datafile belongs to this replic ,why? > > > > # ls > > > /data/dataspace/1/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_3717620888497075523_106232* > > ls: cannot access > > > /data/dataspace/1/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_3717620888497075523_106232*: > > No such file or directory > > [root@CHBM224 conf]# ls > > > /data/dataspace/1/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_2504407693800874616_106252* > > ls: cannot access > > > /data/dataspace/1/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_2504407693800874616_106252*: > > No such file or directory > > [root@CHBM224 conf]# ls > > > /data/dataspace/2/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_2504407693800874616_106252* > > ls: cannot access > > > /data/dataspace/2/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_2504407693800874616_106252*: > > No such file or directory > > [root@CHBM224 conf]# ls > > > /data/dataspace/3/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_2504407693800874616_106252* > > ls: cannot access > > > /data/dataspace/3/current/BP-50684181-192.168.10.220-1383638483950/current/finalized/blk_2504407693800874616_106252*: > > No such file or directory > > [root@CHBM224 conf]# hdfs fsck /alex/terasort/1G-input/part-m-00000 > -files > > -blocks -locations > > Connecting to namenode via http://CHBM220:50070 > > FSCK started by root (auth:SIMPLE) from /192.168.10.224 for path > > /alex/terasort/1G-input/part-m-00000 at Wed Dec 11 11:35:42 CST 2013 > > /alex/terasort/1G-input/part-m-00000 100000000 bytes, 2 block(s): OK > > 0. > BP-50684181-192.168.10.220-1383638483950:blk_3717620888497075523_106232 > > len=67108864 repl=3 [192.168.10.222:50010, 192.168.10.223:50010, > > 192.168.10.221:50010] > > 1. > BP-50684181-192.168.10.220-1383638483950:blk_2504407693800874616_106252 > > len=32891136 repl=3 [192.168.10.222:50010, 192.168.10.221:50010, > > 192.168.10.224:50010] > > > > -- > Harsh J > --047d7bf0ca7c186cfc04ed3eb0bb Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

you are right ,but i only find meta file why no block=A0data file?=
# find /data -name 'blk_-5451264646515882190_106793*'
/data= /dataspace/3/current/BP-50684181-192.168.10.220-1383638483950/current/final= ized/subdir39/blk_-5451264646515882190_106793.meta
=A0
# ls /data/dataspace/3/current/BP-50684181-192.168.10.220-138363848395= 0/current/finalized/subdir39/
blk_3810334848964580951=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0 blk_4621466474283145207_106809.meta=A0 blk_-545126464= 6515882190=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 blk_580162309124277323_10= 6788.meta
blk_3810334848964580951_106801.meta=A0 blk_516060569193828059=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 blk_-5451264646515882190_106793.meta
blk_= 4621466474283145207=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 blk_516060569193= 828059_106796.meta=A0=A0 blk_580162309124277323

=A0
On Wed, Dec 11, 2013 at 3:16 PM, Harsh J <hars= h@cloudera.com> wrote:
Block files are not stored in a flat = directory (to avoid FS limits of
max files under a dir). Instead of look= ing for them right under
finalized, issue a "find" query with the pattern instead and you<= br>should be able to spot it.

On Wed, Dec 11, 2013 at 9:10 AM, ch huang <justlooks@gmail.com> wrote:
> = hi,maillist:
> =A0 =A0 =A0 =A0 =A0 =A0 =A0i try to corrupt a block of= a file in my benchmark environment,
> as the following command i find blk_2504407693800874616_106252 ,it'= ;s replica
> on 192.168.10.224 is my target ,but i find all the datad= ir in 192.168.10.224
> ,can not fine the datafile belongs to this rep= lic ,why?
>
> # ls
> /data/dataspace/1/current/BP-50684181-192.168.10.= 220-1383638483950/current/finalized/blk_3717620888497075523_106232*
>= ls: cannot access
> /data/dataspace/1/current/BP-50684181-192.168.10= .220-1383638483950/current/finalized/blk_3717620888497075523_106232*:
> No such file or directory
> [root@CHBM224 conf]# ls
> /dat= a/dataspace/1/current/BP-50684181-192.168.10.220-1383638483950/current/fina= lized/blk_2504407693800874616_106252*
> ls: cannot access
> /da= ta/dataspace/1/current/BP-50684181-192.168.10.220-1383638483950/current/fin= alized/blk_2504407693800874616_106252*:
> No such file or directory
> [root@CHBM224 conf]# ls
> /dat= a/dataspace/2/current/BP-50684181-192.168.10.220-1383638483950/current/fina= lized/blk_2504407693800874616_106252*
> ls: cannot access
> /da= ta/dataspace/2/current/BP-50684181-192.168.10.220-1383638483950/current/fin= alized/blk_2504407693800874616_106252*:
> No such file or directory
> [root@CHBM224 conf]# ls
> /dat= a/dataspace/3/current/BP-50684181-192.168.10.220-1383638483950/current/fina= lized/blk_2504407693800874616_106252*
> ls: cannot access
> /da= ta/dataspace/3/current/BP-50684181-192.168.10.220-1383638483950/current/fin= alized/blk_2504407693800874616_106252*:
> No such file or directory
> [root@CHBM224 conf]# hdfs fsck /alex= /terasort/1G-input/part-m-00000 -files
> -blocks -locations
> C= onnecting to namenode via http://CHBM220:50070
> FSCK started by root (auth:SIMPLE) from /192.168.10.224 for path
> /alex/terasort/= 1G-input/part-m-00000 at Wed Dec 11 11:35:42 CST 2013
> /alex/terasor= t/1G-input/part-m-00000 100000000 bytes, 2 block(s): =A0OK
> 0. BP-50684181-192.168.10.220-1383638483950:blk_3717620888497075523_10= 6232
> len=3D67108864 repl=3D3 [192.168.10.222:50010, 192.168.10.223:50010,
> 192.168.10.= 221:50010]
> 1. BP-50684181-192.168.10.220-1383638483950:blk_2504= 407693800874616_106252
> len=3D32891136 repl=3D3 [192.168.10.222:50010, 192.168.10.221:50010,
> 192.168.10.= 224:50010]



--
Harsh J

--047d7bf0ca7c186cfc04ed3eb0bb--