Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CC54CC4A9 for ; Tue, 5 Jun 2012 08:51:28 +0000 (UTC) Received: (qmail 15945 invoked by uid 500); 5 Jun 2012 08:51:27 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 15887 invoked by uid 500); 5 Jun 2012 08:51:27 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 15875 invoked by uid 99); 5 Jun 2012 08:51:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jun 2012 08:51:26 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of andreina.j@huawei.com designates 58.251.152.66 as permitted sender) Received: from [58.251.152.66] (HELO szxga03-in.huawei.com) (58.251.152.66) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jun 2012 08:51:21 +0000 Received: from huawei.com (szxga03-in [172.24.2.9]) by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0M54000XLYXBJB@szxga03-in.huawei.com> for hdfs-user@hadoop.apache.org; Tue, 05 Jun 2012 16:44:00 +0800 (CST) Received: from szxrg02-dlp.huawei.com ([172.24.2.119]) by szxga03-in.huawei.com (iPlanet Messaging Server 5.2 HotFix 2.14 (built Aug 8 2006)) with ESMTP id <0M54005QQYW3QT@szxga03-in.huawei.com> for hdfs-user@hadoop.apache.org; Tue, 05 Jun 2012 16:43:59 +0800 (CST) Received: from 172.24.2.119 (EHLO szxeml214-edg.china.huawei.com) ([172.24.2.119]) by szxrg02-dlp.huawei.com (MOS 4.1.9-GA FastPath queued) with ESMTP id AJS02977; Tue, 05 Jun 2012 16:43:07 +0800 (CST) Received: from SZXEML411-HUB.china.huawei.com (10.82.67.138) by szxeml214-edg.china.huawei.com (172.24.2.29) with Microsoft SMTP Server (TLS) id 14.1.323.3; Tue, 05 Jun 2012 16:42:52 +0800 Received: from blrprnc08ns (10.18.96.97) by szxeml411-hub.china.huawei.com (10.82.67.138) with Microsoft SMTP Server id 14.1.323.3; Tue, 05 Jun 2012 16:43:05 +0800 Date: Tue, 05 Jun 2012 14:13:03 +0530 From: andreina Subject: Query in the Number of Under Replicated Blocks displayed in UI and fsck Report X-Originating-IP: [10.18.96.97] To: hdfs-user@hadoop.apache.org Reply-to: andreina.j@huawei.com Message-id: <000901cd42f7$391f99a0$ab5ecce0$%j@huawei.com> Organization: htipl MIME-version: 1.0 X-Mailer: Microsoft Office Outlook 12.0 Content-type: multipart/alternative; boundary="Boundary_(ID_v0cJiAPzHrXG6twW6vDXig)" Content-language: en-us Thread-index: Ac1C9zhUpceyRPd3T8WMFoAM+wEmPg== X-CFilter-Loop: Reflected X-Virus-Checked: Checked by ClamAV on apache.org --Boundary_(ID_v0cJiAPzHrXG6twW6vDXig) Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Hi All, I started NN,2 DN with replication Factor 2 and executed the following scenario Step 1: write a file "a.txt" and hflush() ( blk_ts1 is in DN1 and DN2 under RBW folder) Step 2: Delete the blk_ts1 from DN2 in RBW Step 3: Write data to the same file "a.txt" and close the file "a.txt".(Timestamp will be changed due to recovery and blk_ts2 will be in DN1 under finalized) As discussed for the issue hdfs-3493 blk_ts1 in DN2 under RBW will be marked as corrupt and will not be deleted until the block is replicated to another DN. Hence in BlocksMap the blk will be updated as underreplicated. In fsck report Number of underreplicated block will be :1 ( since it gets the data from BlocksMap) But in UI report Number of Underreplicated blockl will be :0 ( Since it gets the data from VolumeMap) Could someone clarify me Why this difference exist Between UI and fsck report and is this an expected behavior? Huawei Technologies India Pvt. Ltd. Level 3,4 &5 Leela Galleria, The Leela Palce 23, Airport Road, Bangalore - 560008 www.huawei.com ---------------------------------------------------------------------------- --------------------------------------------------------- This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! --Boundary_(ID_v0cJiAPzHrXG6twW6vDXig) Content-type: text/html; charset=us-ascii Content-transfer-encoding: 7BIT

Hi All,

                I started NN,2 DN with replication Factor 2 and executed the following scenario

Step 1: write a file “a.txt”  and hflush() ( blk_ts1 is in DN1 and DN2 under RBW folder)

Step 2:  Delete the blk_ts1 from DN2 in RBW

Step 3: Write data to the same file “a.txt” and close the file “a.txt”.(Timestamp will be changed due to recovery and blk_ts2 will be in DN1 under finalized)

 

                As discussed for the issue hdfs-3493 blk_ts1 in DN2 under RBW and will not be deleted until the block is replicated to another DN. Hence in BlocksMap the blk will be updated as underreplicated.

 

In fsck report  Number of underreplicated block will be :1    ( since it gets the data from BlocksMap)

But in UI report  Number of Underreplicated blockl will be :0 ( Since it gets the data from VolumeMap)

 

Could someone clarify me  Why this difference exist Between UI and fsck report and is this an expected behavior?

 


Huawei Technologies India Pvt. Ltd.
Level 3,4 &5 Leela Galleria, The Leela Palce
23, Airport Road, r>
www.huawei.com
-------------------------------------------------------------------------------------------------------------------------------------
This e-mail and its attachments contain confidential information from HUAWEI, which
is intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!

 

--Boundary_(ID_v0cJiAPzHrXG6twW6vDXig)--