hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lohit Vijayarenu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6248) SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded directories
Date Tue, 15 Apr 2014 23:53:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13970218#comment-13970218
] 

Lohit Vijayarenu commented on HDFS-6248:
----------------------------------------

Here is stack trace of SNN before crash.

{noformat}

2014-03-29 18:07:35,380 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered
exception on operation AddOp [length=0, path=/user/foo/bar.txt, replication=3, mtime=1396116335071,
atime=1396116335071, blockSize=536870912, blocks=[], permissions=foo:supergroup:rw-r--r--,
clientName=DFSClient_attempt_1395346107078_146938_m_000041_1_1098354233_1, clientMachine=10.10.10.10,
opCode=OP_ADD, txid=487688396]
java.lang.NullPointerException
     at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:281)
     at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:171)
     at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:90)
     at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
     at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
2014-03-29 18:07:35,622 WARN org.apache.hadoop.hdfs.server.namenode.NameNode: Quota violation
in image for //user/foo (Namespace quota : 1445052 consumed : 1304943) (Diskspace quota :
2199023255552000 consumed : 2199023483200164).
2014-03-29 18:07:36,429 FATAL org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Unknown
error encountered while tailing edits. Shutting down standby NN.
java.io.IOException: Failed to apply edit log operation AddOp [length=0, path=/user/foo/bar.txt,
replication=3, mtime=1396116335071, atime=1396116335071, blockSize=536870912, blocks=[], permissions=foo:supergroup:rw-r--r--,
clientName=DFSClient_attempt_1395346107078_146938_m_000041_1_1098354233_1, clientMachine=10.10.10.10,
opCode=OP_ADD, txid=487688396]: error null
     at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:94)
     at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:174)
     at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:90)
     at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:708)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:227)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:321)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
     at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:456)
     at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:292)
2014-03-29 18:07:36,431 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-03-29 18:07:36,433 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
{noformat}

> SNN crash during replay of FSEditLog of files inside directories having QuotaExceeded
directories 
> --------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-6248
>                 URL: https://issues.apache.org/jira/browse/HDFS-6248
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.0.6-alpha, 2.4.0
>         Environment: NameNode HA setup with Active/Standby using QJM
>            Reporter: Lohit Vijayarenu
>
> We are seeing cases when Secondary NameNode crashes without recovery when it tries to
replay edit log of files which are part of directories which have exceeded Quota. While debugging
we got stack trace but we are still trying to reproduce this and wanted to note this to see
if anyone else had seen this issue already. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message