hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2910) Active NN reports Bad state: BETWEEN_LOG_SEGMENTS when shared edits dir is inaccessible during log roll
Date Tue, 07 Feb 2012 19:39:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202676#comment-13202676
] 

Bikas Saha commented on HDFS-2910:
----------------------------------

Repro steps
1) Start 2 NN's in active standby mode
2) Remove write permissions from shared edits dir
3) Upon log roll triggered by standby, the active gets error when finalizing the edit logs
4) FSEditLog.rollEditLogs() call endCurrentLogSegment() that hits exception. This exception
is silently ignored assuming logSync will fix it later on
5) FSEditLog.rollEditLogs() now calls startLogSegment() that again hits exception. This exception
is not silently ignored but is caught way up the stack.
6) After that shared edits dir comes back online by giving write permissions
7) Standby triggers log roll again. But active NN keeps complaining about BETWEEN_LOG_SEGMENTS
with the following exception reported on the standby
12/02/07 00:46:55 INFO ha.EditLogTailer: Triggering log roll on remote NameNode localhost/127.0.0.1:28000
12/02/07 00:47:53 WARN ha.EditLogTailer: Unable to trigger a roll of the active NN
java.lang.IllegalStateException: java.lang.IllegalStateException: Bad state: BETWEEN_LOG_SEGMENTS
	at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:887)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.rollEditLog(FSEditLog.java:831)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.rollEditLog(FSImage.java:975)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.rollEditLog(FSNamesystem.java:4026)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rollEditLog(NameNodeRpcServer.java:741)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolServerSideTranslatorPB.rollEditLog(NamenodeProtocolServerSideTranslatorPB.java:116)
	at org.apache.hadoop.hdfs.protocol.proto.NamenodeProtocolProtos$NamenodeProtocolService$2.callBlockingMethod(NamenodeProtocolProtos.java:8068)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:439)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:878)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1608)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1604)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1602)

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:203)
	at $Proxy11.rollEditLog(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.rollEditLog(NamenodeProtocolTranslatorPB.java:162)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.triggerActiveLogRoll(EditLogTailer.java:256)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$3(EditLogTailer.java:253)
	at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:287)
                
> Active NN reports Bad state: BETWEEN_LOG_SEGMENTS when shared edits dir is inaccessible
during log roll
> -------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2910
>                 URL: https://issues.apache.org/jira/browse/HDFS-2910
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message