hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: Edits log apply performance
Date Mon, 19 Jan 2015 17:48:41 GMT
Hi Daniel,

Unfortunately, there likely isn't anything you can do to speed this up.
The process of applying edits will be bound by the available read
throughput on the edits stream and the CPU needs for processing each
transaction.  This is inherently a single-threaded process in the current
implementation.

You can check progress by accessing the NameNode web UI and looking at the
Startup Progress tab.  This won't tell you exactly when it will be done,
but the stats on the page will give you a sense for how quickly it is
making progress.

For the future, you might want to consider deploying monitoring to check
the time of the last checkpoint, and alert if it goes beyond some
threshold.  This might give you a chance to react and fix it before the
next NameNode restart.

There are a few improvements planned in this area for increasing
performance and reducing the chance of running without a recent
checkpoint.  If you're interested, you can watch the following issues in
jira.

https://issues.apache.org/jira/browse/HDFS-4923

https://issues.apache.org/jira/browse/HDFS-6353

https://issues.apache.org/jira/browse/HDFS-7609

Chris Nauroth
Hortonworks
http://hortonworks.com/


On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <danielrulez@gmail.com> wrote:

> Hi,
> After restarting the namenode we discovered that there was no checkpoint
> for quite a while.
> We are waiting for all the changes to be applied to the fsimage, but it
> seems like it will take hours.
>
> Is there something we can do to expedite the process? Increases
> parallelism? Something at all?
>
> Thanks,
> Daniel
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message