hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2339) BackUpNode is not getting shutdown/recover when all volumes failed
Date Fri, 16 Sep 2011 14:38:09 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13106098#comment-13106098
] 

Uma Maheswara Rao G commented on HDFS-2339:
-------------------------------------------

Some more info:

11/09/16 19:31:24 INFO namenode.FSEditLog: Number of transactions: 6 Total time for transactions(ms):
2Number of transactions batched in Syncs: 19 Number of syncs: 6 SyncTimes(ms): 77
11/09/16 19:32:45 INFO namenode.FSEditLog: Number of transactions: 8 Total time for transactions(ms):
2Number of transactions batched in Syncs: 19 Number of syncs: 8 SyncTimes(ms): 77
11/09/16 19:34:42 INFO namenode.FSEditLog: Number of transactions: 10 Total time for transactions(ms):
60401Number of transactions batched in Syncs: 19 Number of syncs: 10 SyncTimes(ms): 77
11/09/16 19:35:40 INFO namenode.FSImage: NameNode started a new log segment at txid 137
11/09/16 19:35:40 INFO namenode.FSEditLog: Ending log segment 121
11/09/16 19:35:40 INFO namenode.FSEditLog: Number of transactions: 11 Total time for transactions(ms):
61080Number of transactions batched in Syncs: 19 Number of syncs: 12 SyncTimes(ms): 77
11/09/16 19:35:40 ERROR namenode.FSEditLog: Error ending log segment (journal JournalAndStream(mgr=FileJournalManager(root=/home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08),
stream=/home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08/current/edits_inprogress_0000000000000000121))
java.io.IOException: Unable to finalize edits file /home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08/current/edits_inprogress_0000000000000000121
        at org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:97)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog$JournalAndStream.close(FSEditLog.java:1209)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog$JournalAndStream.access$4(FSEditLog.java:1202)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog$4.apply(FSEditLog.java:880)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.mapJournalsAndReportErrors(FSEditLog.java:1049)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:876)
        at org.apache.hadoop.hdfs.server.namenode.BackupImage.namenodeStartedLogSegment(BackupImage.java:355)
        at org.apache.hadoop.hdfs.server.namenode.BackupNode$BackupNodeRpcServer.startLogSegment(BackupNode.java:257)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:632)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
11/09/16 19:35:40 ERROR namenode.FSEditLog: Disabling journal 
...............
...............
JournalAndStream(mgr=FileJournalManager(root=/home/Uma/Hadoop-0.24-09162011/hadoop-hdfs-0.24.0-SNAPSHOT/hadoop-root/dfs/name08),
stream=null)
11/09/16 19:35:41 INFO ipc.Server: IPC Server handler 0 on 50100, call: startLogSegment(NamenodeRegistration(HOST-10-18-52-222:9000,
role=NameNode), 137), rpc version=2, client version=1, methodsFingerPrint=-852377201 from
10.18.52.222:43158, error:
java.io.IOException: Unable to start log segment 137: no journals successfully started.
        at org.apache.hadoop.hdfs.server.namenode.FSEditLog.startLogSegment(FSEditLog.java:843)
        at org.apache.hadoop.hdfs.server.namenode.BackupImage.namenodeStartedLogSegment(BackupImage.java:370)
        at org.apache.hadoop.hdfs.server.namenode.BackupNode$BackupNodeRpcServer.startLogSegment(BackupNode.java:257)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:632)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1484)
11/09/16 19:35:41 INFO namenode.FSImage: Waiting until the NameNode rolls its edit logs in
order to freeze the BackupNode namespace.



Dump:

 "JDWP Command Reader" daemon prio=10 tid=0x000000004062e800 nid=0x699d runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Thread-22" daemon prio=10 tid=0x00000000402ad000 nid=0x699a in Object.wait() [0x00007f346f9f8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f348b1007e8> (a org.apache.hadoop.hdfs.server.namenode.BackupImage)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.hadoop.hdfs.server.namenode.BackupImage.waitUntilNamespaceFrozen(BackupImage.java:408)
        - locked <0x00007f348b1007e8> (a org.apache.hadoop.hdfs.server.namenode.BackupImage)
        at org.apache.hadoop.hdfs.server.namenode.Checkpointer.doCheckpoint(Checkpointer.java:200)
        at org.apache.hadoop.hdfs.server.namenode.Checkpointer.run(Checkpointer.java:149)


> BackUpNode is not getting shutdown/recover when all volumes failed
> ------------------------------------------------------------------
>
>                 Key: HDFS-2339
>                 URL: https://issues.apache.org/jira/browse/HDFS-2339
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.24.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>
> When all volumes failed at back up node side, it is waiting for ever.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message