hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cooper Bethea <co...@siftscience.com>
Subject Re: persistent under-replicated blocks
Date Thu, 09 Jan 2014 18:46:58 GMT
I have only 9 under-replicated blocks on the cluster, and it is very
important that I restore my cluster to a fully-replicated state. Is there a
way I can manually copy these blocks to other datanodes, or perhaps new
datanodes?


On Thu, Jan 9, 2014 at 10:34 AM, Cooper Bethea <coops@siftscience.com>wrote:

> Chris, Steve, thanks for responding.
>
> Overnight I ran a script to bump replication, then lower it, as Chris
> suggested. There has been no effect--all underreplicated blocks still have
> only 1 replica.
>
> Steve, I am running the rebalancer.
>
>
> On Thu, Jan 9, 2014 at 1:33 AM, Steve Loughran <stevel@hortonworks.com>wrote:
>
>> are you  running the rebalancer?
>>
>>
>> On 9 January 2014 04:40, Chris Embree <cembree@gmail.com> wrote:
>>
>> > It's too bad that this hasn't been corrected in HDFS 2.0....  I have a
>> > script that I run several times a day to ensure that blocks are
>> replicated
>> > correctly.  Here a link to an article about it:
>> > http://dataforprofit.com/?p=427
>> >
>> >
>> > On Wed, Jan 8, 2014 at 9:00 PM, Cooper Bethea <coops@siftscience.com>
>> > wrote:
>> >
>> > > Following on--is there a way that I can forcibly replicate these
>> blocks,
>> > > perhaps by rsyncing the underlying files to other datanodes? As you
>> might
>> > > imagine under-replicated data makes me very uneasy.
>> > >
>> > >
>> > > On Wed, Jan 8, 2014 at 12:00 PM, Cooper Bethea <coops@siftscience.com
>> > > >wrote:
>> > >
>> > > > Hi HDFS developers,
>> > > >
>> > > > I have a worrying problem in a 2.0.0-cdh4.4.0 HDFS cluster I am
>> > running.
>> > > 9
>> > > > blocks in the cluster are persistently reported to be
>> under-replicated
>> > > per
>> > > > "hdfs fsck".
>> > > >
>> > > > I am able to fetch the files that contain these blocks, so I know
>> that
>> > > the
>> > > > data is there, but for some reason replication is not taking
>> effect. In
>> > > > hopes of getting the cluster to notice that there were
>> under-replicated
>> > > > blocks I tried using "hdfs dfs -setrep" to raise the replication
>> > factor,
>> > > > but the cluster continues to report a single replica for each of
>> these
>> > > > blocks. When viewing master logs I see that the replication factor
>> > change
>> > > > is respected, but there are no messages that refer to the
>> > > under-replicated
>> > > > blocks.
>> > > >
>> > > > Thanks for your time. Please let me know what I can do to
>> investigate
>> > > > further.
>> > > >
>> > >
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message