hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Missing Replicas and Under replicated blocks
Date Tue, 18 Oct 2011 05:09:07 GMT
Stuti,

FSCK, via "hadoop fsck /" would tell you more details on what seems to have caused these missing
replicas. I'm guessing they are probably related to disk corruption (crc failures, etc.),
but would be good to check if its just a few missing block replicas or if your entire set
of replicas itself has been wiped out (leading to a file corruption, which is not good).

So long as you have replication factor set to 3, losses of a couple of replicas to corruption,
unavailability of service, etc. is bearable.

On 18-Oct-2011, at 9:37 AM, Stuti Awasthi wrote:

> Thanks Harsh,
> I will go through what you have suggested. But I think these options will help to prevent
under replicated blocks. What about missing replicas. Is there any way to fix it or to prevent
it ?
> 
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com] 
> Sent: Monday, October 17, 2011 10:58 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Missing Replicas and Under replicated blocks
> 
> Stuti,
> 
> You'll want to:
> 
> 1. Set "2" for mapred.submit.replication in mapred-site.xml to prevent the job related
staging files to be created with default replication factor of 10.
> 2. Learn about the "-setrep" option to toggle replication factors of a path. See http://hadoop.apache.org/common/docs/current/file_system_shell.html#setrep
> 
> HTH.
> 
> On Mon, Oct 17, 2011 at 11:51 AM, Stuti Awasthi <stutiawasthi@hcl.com> wrote:
>> Hi ,
>> 
>> I have a mini Hadoop cluster of 3 nodes with  DN and TT running on each node and
replication is set to 3.By mistake I tried to write few files which requires replication of
10. When I checked the HDFS status with FSCK utility it shows :
>> 
>> /root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/job.jar 2352841 bytes,
1 block(s):  Under replicated blk_-9220069752024407718_1546. Target Replicas is 10 but found
3 replica(s).
>> 0. blk_-9220069752024407718_1546 len=2352841 repl=3 
>> [10.33.100.74:50010, 10.33.64.235:50010, 10.33.100.117:50010]
>> 
>> /root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/libjars 
>> <dir> /root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/libjars/hadoop-0.20.2-core.jar
2707856 bytes, 1 block(s):  Under replicated blk_-4791859336845413240_1544. Target Replicas
is 10 but found 3 replica(s).
>> 0. blk_-4791859336845413240_1544 len=2707856 repl=3 
>> [10.33.100.74:50010, 10.33.64.235:50010, 10.33.100.117:50010] ....................................................
>> Status: HEALTHY
>>  Total size:    45239704 B
>>  Total dirs:    101
>>  Total files:   88 (Files currently being written: 3)
>>  Total blocks (validated):      88 (avg. block size 514087 B) (Total 
>> open file blocks (not validated): 3)
>>  Minimally replicated blocks:   88 (100.0 %)
>>  Over-replicated blocks:        0 (0.0 %)
>>  Under-replicated blocks:       20 (22.727272 %)
>>  Mis-replicated blocks:         0 (0.0 %)
>>  Default replication factor:    2
>>  Average block replication:     2.4318182
>>  Corrupt blocks:                0
>>  Missing replicas:              140 (65.42056 %)
>>  Number of data-nodes:          3
>>  Number of racks:               1
>> 
>> I wanted to know that can I fix :
>>        1.     Under-replicated
>>        2.     Missing replicas
>> 
>> Please Suggest
>> Thanks
>> Stuti
>> 
>> ::DISCLAIMER::
>> ----------------------------------------------------------------------
>> -------------------------------------------------
>> 
>> The contents of this e-mail and any attachment(s) are confidential and intended for
the named recipient(s) only.
>> It shall not attach any liability on the originator or HCL or its 
>> affiliates. Any views or opinions presented in this email are solely those of the
author and may not necessarily reflect the opinions of HCL or its affiliates.
>> Any form of reproduction, dissemination, copying, disclosure, 
>> modification, distribution and / or publication of this message 
>> without the prior written consent of the author of this e-mail is 
>> strictly prohibited. If you have received this email in error please delete it and
notify the sender immediately. Before opening any mail and attachments please check them for
viruses and defect.
>> 
>> ----------------------------------------------------------------------
>> -------------------------------------------------
>> 
> 
> 
> 
> --
> Harsh J


Mime
View raw message