hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Meagher <john.meag...@gmail.com>
Subject Re: Fixing Mis-replicated blocks
Date Fri, 21 Oct 2011 17:24:07 GMT
In this case everything should be 3.  I was hoping there was a quicker
way.  The -w option should help so this doesn't need to be run again.

On Thu, Oct 20, 2011 at 20:26, Jeff Bean <jwfbean@cloudera.com> wrote:
> Do setrep -w on the increase to force the new replica before decreasing
> again.
>
> Of course, the little script only works if the replication factor is 3 on
> all the files. If it's a variable amount you should use the java API to get
> the existing factor and then increase by one and then decrease.
>
> Jeff
>
> On Thu, Oct 20, 2011 at 8:44 AM, John Meagher <john.meagher@gmail.com>wrote:
>
>> After a hardware move with an unfortunate mis-setup rack awareness
>> script our hadoop cluster has a large number of mis-replicated blocks.
>>  After about a week things haven't gotten better on their own.
>>
>> Is there a good way to trigger the name node to fix the mis-replicated
>> blocks?
>>
>> Here's what I'm using for now, but it is very slow:
>> for f in `hadoop fsck / | grep "Replica placement policy is violated"
>> | head -n3000 | awk -F: '{print $1}'`; do
>>    hadoop fs -setrep 4 $f
>>    hadoop fs -setrep 3 $f
>> done
>>
>> John
>>
>

Mime
View raw message