hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tombin <tombin.a...@gmail.com>
Subject New cluster help
Date Thu, 14 Jul 2016 19:59:30 GMT
I am setting up a new hadoop cluster for the first time.  My setup
currently looks as follows:

hdfs cluster:
1 namenode
2 datanodes

hbase:
1 hbase node

zookeeper cluster:
3 zookeeper nodes

I have enabled ssl on the hdfs cluster.  When trying to connect from base i
see the following error:

2016-07-14 19:38:58,333 WARN  [Thread-73] hdfs.DFSClient: DataStreamer
Exception

java.lang.NullPointerException

at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferEncryptor.getEncryptedStreams(DataTransferEncryptor.java:191)

at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1335)

at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1281)

at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:526)

2016-07-14 19:39:04,341 INFO  [hb01:16000.activeMasterManager]
hdfs.DFSClient: Could not complete /hbase/.tmp/hbase.version retrying...



this will repeat several time and then i'll throw the following exception:


2016-07-14 19:39:58,772 FATAL [hb01:16000.activeMasterManager]
master.HMaster: Unhandled exception. Starting shutdown.

java.io.IOException: Unable to close file because the last block does not
have enough number of replicas.

at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2151)

at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2119)

at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)

at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)

at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:730)

at org.apache.hadoop.hbase.util.FSUtils.setVersion(FSUtils.java:705)

at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:662)

at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:462)

at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:153)

at
org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128)

at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:652)

at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:185)

at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1750)

at java.lang.Thread.run(Thread.java:745)


hbase shuts down at this point.


on the datanode i side i see the following in the logs that looks like it
may be related:


2016-07-14 19:38:23,132 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
hd03.domain.com:50010:DataXceiver
error processing unknown operation  src: /10.0.0.10:34893 dst: /
10.0.1.10:50010

java.io.EOFException

        at java.io.DataInputStream.readInt(DataInputStream.java:392)

        at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:358)

        at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)

        at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:110)

        at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:193)

        at java.lang.Thread.run(Thread.java:745)

2016-07-14 19:39:33,575 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
hd03.domain.com:50010:DataXceiver
error processing unknown operation  src: /10.0.0.10:34898 dst: /
10.0.1.10:50010

java.io.EOFException

        at java.io.DataInputStream.readInt(DataInputStream.java:392)

        at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:358)

        at
org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getEncryptedStreams(SaslDataTransferServer.java:178)

Is this in relation to my ssl configuration ?
I'm confused about whats going on here.  Thank you in advance for any help.

Mime
View raw message