hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3921) NN will prematurely consider blocks missing when entering active state while still in safe mode
Date Thu, 08 Nov 2012 00:38:12 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron T. Myers updated HDFS-3921:
---------------------------------

     Target Version/s: 2.0.3-alpha
    Affects Version/s:     (was: 2.0.1-alpha)
                       2.0.2-alpha
             Assignee: Aaron T. Myers
              Summary: NN will prematurely consider blocks missing when entering active state
while still in safe mode  (was: corrupt files web UI hits error: Cannot run listCorruptFileBlocks
because replication queues have not been initialized.)

I've taken a look into this. The trouble is that the NN is always processing its repl queues
when it enters the active state even if it's still in startup safemode, i.e. not all DNs have
reported in. This causes the web UI to prematurely indicate that blocks are missing, when
in fact the NN should still just simply be waiting for more DNs to report in order to exit
startup safemode.
                
> NN will prematurely consider blocks missing when entering active state while still in
safe mode
> -----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3921
>                 URL: https://issues.apache.org/jira/browse/HDFS-3921
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.0.2-alpha
>            Reporter: Stephen Chu
>            Assignee: Aaron T. Myers
>
> I shut down all the HDFS daemons in an Highly Available (automatic failover) cluster.
> Then I started one NN and it transitioned it to active. No DNs were started, and I saw
the red warning link on the NN web UI:
> WARNING : There are 36 missing blocks. Please check the logs or run fsck in order to
identify the missing blocks.
> I clicked this to go to the corrupt_files.jsp page, which ran into the following error:
> {noformat}
> HTTP ERROR 500
> Problem accessing /corrupt_files.jsp. Reason:
>     Cannot run listCorruptFileBlocks because replication queues have not been initialized.
> Caused by:
> java.io.IOException: Cannot run listCorruptFileBlocks because replication queues have
not been initialized.
> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.listCorruptFileBlocks(FSNamesystem.java:5035)
> 	at org.apache.hadoop.hdfs.server.namenode.corrupt_005ffiles_jsp._jspService(corrupt_005ffiles_jsp.java:78)
> 	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> 	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> 	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 	at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1039)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> 	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> 	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> 	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> 	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> 	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> 	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> 	at org.mortbay.jetty.Server.handle(Server.java:326)
> 	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> 	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> 	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> 	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> 	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> 	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> 	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message