Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of chris@rapleaf.com designates
 208.96.16.213 as permitted sender)
DomainKey-Signature: a=rsa-sha1;
 h=Received:Mime-Version:In-Reply-To:References:Content-Type:Message-Id:From:Subject:Date:To:X-Mailer;
 b=Yj09+DGSQSZ0R7m8AGePS9eW0fAwtvnILpmwUlv2eo6YfgbEMTOcjxVmm+q/Fmvns16i03VhqCwAKcga4AM/2LkTGhr3Mu4zlm+6REXTCOUcyLhqXFHu5Kt8TMgfi65+7se2YOeUyFtysxbg2KdiILJr9dfpsWRT2vUsfZKFVnI=;
 c=nofws; d=rapleaf.com; q=dns; s=m1
Mime-Version: 1.0 (Apple Message framework v752.3)
In-Reply-To: <47828C5A.3010800@yahoo-inc.com>
References: <028197BF-37EA-424C-B817-043AAD444FB7@rapleaf.com>
 <477E9EE4.2030005@yahoo-inc.com>
 <F579C054-46DB-44A1-9974-12F0DD965C71@rapleaf.com>
 <47828C5A.3010800@yahoo-inc.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-1-274224241
Message-Id: <6FE30D17-69A4-4B8F-897C-D0394BE98913@rapleaf.com>
From: Chris Kline <chris@rapleaf.com>
Subject: Re: Under replicated block doesn't get fixed until DFS restart
Date: Mon, 7 Jan 2008 22:03:59 -0800
To: hadoop-user@lucene.apache.org

--Apple-Mail-1-274224241
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=US-ASCII;
	delsp=yes;
	format=flowed

Sure.  Here are 2 blocks.

----- blk_4522585614366970680

hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-02:2008-01-02  
17:06:17,058 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.allocateBlock: /data/hbase1/hregion_70236052/ 
compaction.dir/hregion_70236052/info/mapfiles/-1/index.  
blk_4522585614366970680
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-02:2008-01-02  
17:06:17,062 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.addStoredBlock: blockMap updated: 10.100.11.31:7277 is  
added to blk_4522585614366970680

------ Note that the following is AFTER I restarted HDFS -------

hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:15,882 DEBUG org.apache.hadoop.dfs.StateChange:  
UnderReplicationBlocks.update blk_4522585614366970680 curReplicas 1  
curExpectedReplicas 3 oldReplicas 0 oldExpectedReplicas  3 curPri  0  
oldPri  3
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:15,882 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.UnderReplicationBlock.update:blk_4522585614366970680 has  
only 1 replicas and need 3 replicas so is added to neededReplications  
at priority level 0
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:58,808 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.UnderReplicationBlock.remove: Removing block  
blk_4522585614366970680 from priority queue 0
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:58,808 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.pendingTransfer: blk_4522585614366970680 is removed from  
neededReplications to pendingReplications
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:58,808 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.pendingTransfer: ask 10.100.11.31:7277 to replicate  
blk_4522585614366970680 to datanode(s) 10.100.11.59:7277  
10.100.11.64:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:59,716 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.blockReceived: blk_4522585614366970680 is received from  
10.100.11.59:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:59,716 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.addStoredBlock: blockMap updated: 10.100.11.59:7277 is  
added to blk_4522585614366970680
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:59,717 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.blockReceived: blk_4522585614366970680 is received from  
10.100.11.64:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:59,717 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.addStoredBlock: blockMap updated: 10.100.11.64:7277 is  
added to blk_4522585614366970680

------ blk_1984980330938654629

hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-02:2008-01-02  
17:11:17,231 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.allocateBlock: /data/hbase1/hregion_70236052/ 
compaction.dir/hregion_70236052/info/done. blk_1984980330938654629
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-02:2008-01-02  
17:11:17,234 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.addStoredBlock: blockMap updated: 10.100.11.31:7277 is  
added to blk_1984980330938654629

------ Note that the following is AFTER I restarted HDFS -------

hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:15,856 DEBUG org.apache.hadoop.dfs.StateChange:  
UnderReplicationBlocks.update blk_1984980330938654629 curReplicas 1  
curExpectedReplicas 3 oldReplicas 0 oldExpectedReplicas  3 curPri  0  
oldPri  3
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:15,856 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.UnderReplicationBlock.update:blk_1984980330938654629 has  
only 1 replicas and need 3 replicas so is added to neededReplications  
at priority level 0
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:55,805 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.UnderReplicationBlock.remove: Removing block  
blk_1984980330938654629 from priority queue 0
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:55,805 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.pendingTransfer: blk_1984980330938654629 is removed from  
neededReplications to pendingReplications
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:55,805 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.pendingTransfer: ask 10.100.11.31:7277 to replicate  
blk_1984980330938654629 to datanode(s) 10.100.11.62:7277  
10.100.11.61:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:56,660 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.blockReceived: blk_1984980330938654629 is received from  
10.100.11.62:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:56,660 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.addStoredBlock: blockMap updated: 10.100.11.62:7277 is  
added to blk_1984980330938654629
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:56,663 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.blockReceived: blk_1984980330938654629 is received from  
10.100.11.61:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03  
16:17:56,663 INFO org.apache.hadoop.dfs.StateChange: BLOCK*  
NameSystem.addStoredBlock: blockMap updated: 10.100.11.61:7277 is  
added to blk_1984980330938654629

On Jan 7, 2008, at 12:32 PM, Raghu Angadi wrote:

>
> Could you grep for one of the block ids (say 4522585614366970680)  
> in the  namenode logs and post them here with timestamps?
>
> thanks,
> Raghu.
>
> Chris Kline wrote:
>> I believe there was at least one good block (see fsck output).   
>> All data nodes were up at the time according to the web page.  I  
>> grep'd the namenode log files for the under replicated blocks and  
>> only got an entry for when it was created and entries for when the  
>> replication was fixed after the HDFS restart.  Here is the result  
>> of fsck:
>> $HADOOP_HOME/bin/hadoop fsck /
>> .......................................................
>> /data/hbase1/hregion_70236052/compaction.dir/hregion_70236052/info/ 
>> done:  Under replicated blk_1984980330938654629. Target Replicas  
>> is 3 but found 1 replica(s).
>> ..
>> /data/hbase1/hregion_70236052/info/info/2807320534360768620:   
>> Under replicated blk_1717622121416314549. Target Replicas is 3 but  
>> found 1 replica(s).
>> .
>> /data/hbase1/hregion_70236052/info/mapfiles/2807320534360768620/ 
>> data:  Under replicated blk_-5019714262388221150. Target Replicas  
>> is 3 but found 1 replica(s).
>> .
>> /data/hbase1/hregion_70236052/info/mapfiles/2807320534360768620/ 
>> index:  Under replicated blk_4522585614366970680. Target Replicas  
>> is 3 but found 1 replica(s).
>> .........................................
>> /data/hbase1/log_10.100.11.63_1199307142676_60020/hlog.dat.000:   
>> Under replicated blk_-2871471426720379908. Target Replicas is 3  
>> but found 1 replica(s).
>> .......
>> /data/hbase1/log_10.100.11.65_1199307142711_60020/hlog.dat.000:  
>> MISSING 1 blocks of total size 0 B.
>> .Status: CORRUPT
>>  Total size:    71009158262 B
>>  Total blocks:  16318 (avg. block size 4351584 B)
>>  Total dirs:    21416
>>  Total files:   16253
>>  Over-replicated blocks:        0 (0.0 %)
>>  Under-replicated blocks:       5 (0.03064101 %)
>>  Target replication factor:     3
>>  Real replication factor:       2.9993873
>> The filesystem under path '/' is CORRUPT
>> -Chris
>> On Jan 4, 2008, at 1:02 PM, Raghu Angadi wrote:
>>> This is of course not expected. A more detailed info or log  
>>> message would help. Do you know if there is at least one good  
>>> block? Sometimes, the remaining "good" block might actually be  
>>> corrupted and thus can not replicate itself. Restarting might  
>>> just have brought up the datanodes that were down (for whatever  
>>> reason) before the restart.
>>>
>>> Raghu.
>>>
>>> Chris Kline wrote:
>>>> fsck reports several under replicated blocks, but these do not  
>>>> get fixed until I restart DFS.  fsck also reports a missing  
>>>> block at the same time, but this should affect the function of  
>>>> fixing under replicated blocks.  Has anyone seen this before?
>>>> I'm running 0.15.0.
>>>> -Chris Kline
>>>
>> -Chris
>> We're hiring engineers.  $10,007 reward for referrals we hire.
>

-Chris

We're hiring engineers.  $10,007 reward for referrals we hire.


--Apple-Mail-1-274224241--