hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: DataBlockScanner's rate limit
Date Fri, 02 Aug 2013 03:27:33 GMT

The DataBlockScanner isn't responsible for the DN block reports at
startup, which is a wholly different thread/process - it is a NN
independent operation that merely verifies blocks in the background
for the DN's own health. Depending on what the outage caused, it is
likely that you are missing a mount and perhaps blocks of files with a
single replica. Run an fsck to identify what files these are and if
they used a single replication factor?

On Fri, Aug 2, 2013 at 7:25 AM, Davey Yan <davey.yan@gmail.com> wrote:
> I recently got a mini cluster corrupted after my inappropriate process.
> This mini cluster's dfs.replication was set to 1.
> After irregular restart of OS, I cannot wait to leave safemode, the block
> ratio is 0.9862, < 0.999.
> In the http://ip:50075/blockScannerReport, I notice there is rate limit to
> 1MB.
> It will verify the blocks for long time.
> So I "hadoop dfdsadmin safemode leave", and then I got blocks missing.
> My question is: Why should we limit the rate in DataBlockScanner while the
> cluster is still starting up or still in safemode?
> I read the source code of DataBlockScanner.java, there is no parameter to
> change the rate limit.
> It seams to be 1MB to 8MB always.
> --
> Davey Yan

Harsh J

View raw message