hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haijun Cao" <hai...@kindsight.net>
Subject RE: setrep
Date Wed, 04 Jun 2008 06:57:22 GMT


Thanks for the explanation. If that's the case, then it is not slower than expected.  


-----Original Message-----
From: lohit [mailto:lohit_bv@yahoo.com]
Sent: Wed 6/4/2008 2:11 AM
To: core-user@hadoop.apache.org
Subject: Re: setrep

>It seems that setrep won't force replication change to the specified number immediately,
it changed really slowly. just wondering if this is the expected behavior? what's the rational
for this behavior? is there way to speed it up? 

Yes, it wont force replication to be instant. Once you increase the replication factor of
a file, namenode adds it to neededReplication list. Namenode has a thread running which periodically
scans this list and chooses a set of blocks which are under replicated and request the datanodes
to replicate them. This interval is configured using dfs.replication.interval config variable.
Interval dfs.replication.interval is in seconds.
The list of neededReplication also maintains a priority policy, where in blocks with only
one copy would be replicated first. 

If you do not have lot of underReplicated blocks, this should happen pretty fast. Are you
seeing very long delays? 

  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message