hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7609) startup used too much time to load edits
Date Thu, 12 Feb 2015 20:36:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318940#comment-14318940

Chris Nauroth commented on HDFS-7609:

I tried testing an old client without call ID support connecting to a NameNode with retry
cache support.  That just fails fast though due to violating the protobuf spec for the message.
 (I should have expected that.)  That rules out that theory.

The edit log replaying code looks to be correct for the case of old edits with a layout version
before the RPC IDs were logged.  Even if that was a problem, it would just be a one-time slowdown
during an upgrade, not an ongoing problem.

My next theory is that perhaps we have another case of thread-local storage biting us.  HDFS-7385
reported edit logs containing incorrect ACLs, and the root cause was that we had failed to
reinitialize the thread-local op instance completely each time we used it.  Perhaps we have
a similar situation here, where an op instance is getting logged, but some code path failed
to update the op with a unique client ID + call ID.  If that's the case, then HDFS-7398 might
help.  That patch guarantees the full state of the op gets reset on each use, including the
RPC IDs.  I still don't have a repro of my own though, so I can't confirm or deny the theory.

If anyone has an edit log exhibiting the problem that they'd be willing to share, that would
be a big help.  I'd be interested in seeing if there is any pattern to the kinds of ops that
are hitting duplicate RPC IDs or the actual duplicate RPC ID values.  That could help narrow
the problem down to a specific code path.

> startup used too much time to load edits
> ----------------------------------------
>                 Key: HDFS-7609
>                 URL: https://issues.apache.org/jira/browse/HDFS-7609
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.2.0
>            Reporter: Carrey Zhan
>         Attachments: HDFS-7609-CreateEditsLogWithRPCIDs.patch, recovery_do_not_use_retrycache.patch
> One day my namenode crashed because of two journal node timed out at the same time under
very high load, leaving behind about 100 million transactions in edits log.(I still have no
idea why they were not rolled into fsimage.)
> I tryed to restart namenode, but it showed that almost 20 hours would be needed before
finish, and it was loading fsedits most of the time. I also tryed to restart namenode in recover
mode, the loading speed had no different.
> I looked into the stack trace, judged that it is caused by the retry cache. So I set
dfs.namenode.enable.retrycache to false, the restart process finished in half an hour.
> I think the retry cached is useless during startup, at least during recover process.

This message was sent by Atlassian JIRA

View raw message