hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <danielru...@gmail.com>
Subject Re: Edits log apply performance
Date Mon, 19 Jan 2015 18:10:46 GMT
Thought so...
Thank you for the info and assistance.

BR,
Daniel

> On 19 בינו׳ 2015, at 19:48, Chris Nauroth <cnauroth@hortonworks.com> wrote:
> 
> Hi Daniel,
> 
> Unfortunately, there likely isn't anything you can do to speed this up.  The process
of applying edits will be bound by the available read throughput on the edits stream and the
CPU needs for processing each transaction.  This is inherently a single-threaded process in
the current implementation.
> 
> You can check progress by accessing the NameNode web UI and looking at the Startup Progress
tab.  This won't tell you exactly when it will be done, but the stats on the page will give
you a sense for how quickly it is making progress.
> 
> For the future, you might want to consider deploying monitoring to check the time of
the last checkpoint, and alert if it goes beyond some threshold.  This might give you a chance
to react and fix it before the next NameNode restart.
> 
> There are a few improvements planned in this area for increasing performance and reducing
the chance of running without a recent checkpoint.  If you're interested, you can watch the
following issues in jira.
> 
> https://issues.apache.org/jira/browse/HDFS-4923
> 
> https://issues.apache.org/jira/browse/HDFS-6353
> 
> https://issues.apache.org/jira/browse/HDFS-7609
> 
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> 
> 
>> On Sat, Jan 17, 2015 at 9:17 AM, Daniel Haviv <danielrulez@gmail.com> wrote:
>> Hi,
>> After restarting the namenode we discovered that there was no checkpoint for quite
a while.
>> We are waiting for all the changes to be applied to the fsimage, but it seems like
it will take hours.
>> 
>> Is there something we can do to expedite the process? Increases parallelism? Something
at all?
>> 
>> Thanks,
>> Daniel
> 
> 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential, privileged and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any printing, copying, dissemination, distribution, disclosure
or forwarding of this communication is strictly prohibited. If you have received this communication
in error, please contact the sender immediately and delete it from your system. Thank You.

Mime
View raw message