hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9746) Some Kerberos related tests intermittently fails.
Date Wed, 03 Feb 2016 00:42:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129461#comment-15129461 ] 

Xiao Chen commented on HDFS-9746:
---------------------------------

Log excerpt of the failure I saw in {{TestSecureNNWithQJM#testSecureMode}}:
Error Message
{noformat}
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0.  Expected transaction ID was 1
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
{noformat}
stdout
{noformat}
2016-02-01 06:41:47,944 INFO  namenode.FSImage (FSImage.java:loadEdits(832)) - Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@12e8c783 expecting start txid #1
2016-02-01 06:41:47,944 INFO  namenode.FSImage (FSEditLogLoader.java:loadFSEdits(139)) - Start loading edits file https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
2016-02-01 06:41:47,945 INFO  namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID, https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
2016-02-01 06:41:47,945 INFO  namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
2016-02-01 06:41:48,131 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46417 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
	... 15 more
2016-02-01 06:41:48,133 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46417 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 60 DF 82 D5 14 DB 78 8D A8 E4 6D F8 FE 3A F4 AB 98 25 9D DB 51 ED 3B CE 53 C8 DC 48 1C CB EB B5 1B 5A 45 BA CD 68 0A 26 2F 8D 3A FE 75 AE 36 4B 25 B5 B8 5A C1 27 71 E3 B6 03 7D D6 2D 14 58 CD 6D 19 F0 25 D0 5A 9B 35 A6 7E 36 62 DA 28 56 0B E9 53 03 43 7B 71 D5 ED 8F 52 CE 6E 8A 23 0C 52 53 EB 42 0B 7A 6B 8C 54 EB 1C 70 FB 21 DD DF 23 B4 5E AD 42 67 65 42 61 FD DB 2D 28 C2 4D 7A 71 69 D0 74 9A 64 8A 82 A0 EC C8 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 6B 88 55 A5 A8 14 30 12 02 01 12 02 01 11 02 01 10)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
	... 15 more
2016-02-01 06:41:48,138 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46418 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
	... 15 more
2016-02-01 06:41:48,139 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46418 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A 7E 36 73 CB EC 41 15 69 53 02 B8 E4 28 1A 8D 61 56 3E 30 4E 0F C6 28 7C 12 CF EB 25 1E A9 3B 28 99 68 3B CE F3 38 50 B0 DA D9 0F 25 3A 39 B4 12 39 44 A6 64 8D FA 1B 62 22 27 7E 64 D8 C4 FE D3 89 DC 82 E2 C2 21 7E 14 BE DE 89 C6 9D FD C5 3F 25 40 9A F7 2C 26 DD 16 E1 EF C5 66 A5 AD 1A 46 8E 44 9C 86 63 9B 80 BA 81 45 78 97 F4 C1 EF EF 05 68 77 2B 61 B2 DF 30 0B BF E1 94 69 44 2C 97 7D 54 52 F8 9C 88 14 56 A0 B7 A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 3F 0B C9 19 A8 14 30 12 02 01 12 02 01 11 02 01 10)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
	... 15 more
2016-02-01 06:41:48,139 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
2016-02-01 06:41:48,140 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445)
	at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
	at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470)
	... 45 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
	... 49 more
Caused by: java.io.IOException: Cannot get a KDC reply
	at sun.security.krb5.KdcComm.send(KdcComm.java:250)
	at sun.security.krb5.KdcComm.send(KdcComm.java:191)
	at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186)
	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
	... 56 more
2016-02-01 06:41:48,141 ERROR namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(221)) - Got error reading edit log input stream https://localhost:55326/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID; failing over to edit log https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
2016-02-01 06:41:48,142 INFO  namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) - Fast-forwarding stream 'https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID' to transaction ID 1
2016-02-01 06:41:48,169 ERROR protocol.KerberosProtocolHandler (KerberosProtocolHandler.java:exceptionCaught(157)) - /127.0.0.1:46420 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
	... 15 more
2016-02-01 06:41:48,169 ERROR server.KERBEROS_LOG (KerberosProtocolHandler.java:exceptionCaught(158)) - /127.0.0.1:46420 EXCEPTION
org.apache.mina.filter.codec.ProtocolDecoderException: java.lang.NullPointerException: message (Hexdump: 00 00 02 45 6C 82 02 41 30 82 02 3D A1 03 02 01 05 A2 03 02 01 0C A3 82 01 C4 30 82 01 C0 30 82 01 BC A1 03 02 01 01 A2 82 01 B3 04 82 01 AF 6E 82 01 AB 30 82 01 A7 A0 03 02 01 05 A1 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 81 F6 61 81 F3 30 81 F0 A0 03 02 01 05 A1 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A2 20 30 1E A0 03 02 01 02 A1 17 30 15 1B 06 6B 72 62 74 67 74 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 81 B7 30 81 B4 A0 03 02 01 11 A2 81 AC 04 81 A9 6D 55 CA 9E E9 6A 4D 9F C1 47 9A 2B 9E 75 07 73 F4 48 4A 36 AE BD 28 4D DA 3D 00 89 0D B1 70 C1 67 E7 44 0B C4 3E BF 59 3D 2B F2 EA E7 09 05 44 23 2D B6 D6 76 46 D8 DC 05 32 68 A2 2B D5 58 3D EC 5D DD 4D A1 6D F2 80 95 5F 61 2A 40 C5 D3 F7 BD A3 71 73 F2 81 DD CD B1 B3 1D 8E FA B0 70 9F 88 AD 97 7C 2A 91 DE 4F 69 5A 23 17 AA 21 99 7E 89 36 E6 1A 01 06 D2 DF 6F C3 76 15 47 81 9A 65 66 F7 CC 23 52 C5 DE 77 09 AD 66 38 94 DE 93 DC CA 24 6B C4 2B FA 1A BC FC 07 84 9B CE 0D 15 BA C7 00 8A 5C 0E 61 D8 BE 01 A4 81 98 30 81 95 A0 03 02 01 11 A2 81 8D 04 81 8A BA 11 25 9F 01 D1 21 8A 31 30 5B 14 7A 6D A5 65 C1 87 95 F1 86 3A E1 A9 7E 3E 7F 4D FE 8E 8F 96 E4 0A 78 2B 5F 0D 1D 87 88 51 A4 5B 92 94 6A FC 87 97 2A 9C 69 81 B3 5D 5D 67 56 E9 97 97 5D 29 7B B9 B8 9E 99 8F 7A BD 3A E3 4D 6E BA 55 0C ED 3A 67 0F A9 45 B5 93 9B 48 FF FB 45 A7 23 5A B9 52 44 E4 54 D5 08 D5 2F FF 4A C4 52 16 FA 5F 1C 62 41 88 15 C1 B6 63 13 DA 16 33 56 63 37 09 63 2E 1D C8 42 55 FB FA 33 9B CC A4 69 30 67 A0 07 03 05 00 00 00 00 00 A2 0D 1B 0B 45 58 41 4D 50 4C 45 2E 43 4F 4D A3 1C 30 1A A0 03 02 01 00 A1 13 30 11 1B 04 48 54 54 50 1B 09 6C 6F 63 61 6C 68 6F 73 74 A5 11 18 0F 31 39 37 30 30 31 30 31 30 30 30 30 30 30 5A A7 06 02 04 01 AA D4 AF A8 14 30 12 02 01 12 02 01 11 02 01 10)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:234)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:48)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:802)
	at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:120)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
	at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:426)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:604)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:564)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:553)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$400(AbstractPollingIoProcessor.java:57)
	at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:892)
	at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: message
	at org.apache.mina.filter.codec.AbstractProtocolDecoderOutput.write(AbstractProtocolDecoderOutput.java:44)
	at org.apache.directory.server.kerberos.protocol.codec.MinaKerberosDecoder.decode(MinaKerberosDecoder.java:65)
	at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:224)
	... 15 more
2016-02-01 06:41:48,170 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1696)) - PriviledgedActionException as:slave/localhost@EXAMPLE.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
2016-02-01 06:41:48,170 ERROR namenode.EditLogInputStream (EditLogFileInputStream.java:nextOpImpl(194)) - caught exception initializing https://localhost:48398/getJournal?jid=myjournal&segmentTxId=1&storageInfo=-60%3A1226055383%3A0%3AtestClusterID
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:473)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:465)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:445)
	at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:439)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:464)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:306)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:196)
	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
	at org.apache.hadoop.hdfs.web.URLConnectionFactory.openConnection(URLConnectionFactory.java:161)
	at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:470)
	... 45 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Cannot get a KDC reply)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:715)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:285)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:261)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:261)
	... 49 more
Caused by: java.io.IOException: Cannot get a KDC reply
	at sun.security.krb5.KdcComm.send(KdcComm.java:250)
	at sun.security.krb5.KdcComm.send(KdcComm.java:191)
	at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:186)
	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:201)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
	... 56 more
2016-02-01 06:41:48,171 ERROR namenode.FSImage (FSEditLogLoader.java:loadEditRecords(196)) - Error replaying edit log at offset 0.  Expected transaction ID was 1
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
2016-02-01 06:41:48,173 WARN  namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(769)) - Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.namenode.EditLogInputException: Error replaying edit log at offset 0.  Expected transaction ID was 1
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:200)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:140)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:835)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:690)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:281)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1862)
	at org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1827)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.restartNameNode(TestSecureNNWithQJM.java:197)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.doNNWithQJMTest(TestSecureNNWithQJM.java:179)
	at org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM.testSecureMode(TestSecureNNWithQJM.java:159)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up to 5
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
	at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
	at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:187)
	... 27 more
2016-02-01 06:41:48,187 INFO  mortbay.log (Slf4jLog.java:info(67)) - Stopped SslSocketConnectorSecure@localhost:41570
2016-02-01 06:41:48,287 INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1744)) - Shutting down the Mini HDFS Cluster
{noformat}

> Some Kerberos related tests intermittently fails.
> -------------------------------------------------
>
>                 Key: HDFS-9746
>                 URL: https://issues.apache.org/jira/browse/HDFS-9746
>             Project: Hadoop HDFS
>          Issue Type: Test
>            Reporter: Xiao Chen
>            Assignee: Xiao Chen
>
> So far I've seen {{TestSecureNNWithQJM#testSecureMode}} and {{TestKMS#testACLs}} failing. More details coming in the 1st comment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message