hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arulanand Dayalan <arul_anand_2...@yahoo.com.INVALID>
Subject Hadoop Security - NN fails to connect to JN on different servers
Date Wed, 25 May 2016 18:11:23 GMT
Hi,
I had setup Hadoop HA Cluster(QJM) with kerberos. Cluster has three journal nodes. Namenode
is able to connectto journal node running on the Namenode. For other Journal nodes its not
able to connect and throws below exception.Can you please provide inputs for resolving the
issue.
2016-05-25 23:09:47,526 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:/************************************************************STARTUP_MSG:
Starting NameNodeSTARTUP_MSG:   host = server1.cehdev.com/10.67.169.45STARTUP_MSG:   args
= []STARTUP_MSG:   version = 2.5.2STARTUP_MSG:   classpath = STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git
-r cc72e9b000545b86b75a61f4835eb86d57bfafc0; compiled by 'jenkins' on 2014-11-14T23:45ZSTARTUP_MSG:
  java = 1.7.0_79************************************************************/2016-05-25
23:09:47,550 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal
handlers for [TERM, HUP, INT]2016-05-25 23:09:47,557 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
createNameNode []2016-05-25 23:09:48,147 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties2016-05-25 23:09:48,307 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s).2016-05-25 23:09:48,307 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
NameNode metrics system started2016-05-25 23:09:48,309 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
fs.defaultFS is hdfs://hdcluster2016-05-25 23:09:48,310 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
Clients are to use hdcluster to access this namenode/service.2016-05-25 23:09:49,331 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for user hdfs/server1.cehdev.com@CEHDEV.COM
using keytab file /etc/security/keytab/hdfs.service.keytab2016-05-25 23:09:49,392 INFO org.apache.hadoop.hdfs.DFSUtil:
Starting web server as: HTTP/server1.cehdev.com@CEHDEV.COM2016-05-25 23:09:49,392 INFO org.apache.hadoop.hdfs.DFSUtil:
Starting Web-server for hdfs at: http://server1.cehdev.com:500702016-05-25 23:09:49,480 INFO
org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog2016-05-25
23:09:49,487 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.namenode
is not defined2016-05-25 23:09:49,504 INFO org.apache.hadoop.http.HttpServer2: Added global
filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)2016-05-25 23:09:49,510
INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context hdfs2016-05-25 23:09:49,510 INFO org.apache.hadoop.http.HttpServer2: Added filter
static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context logs2016-05-25 23:09:49,511 INFO org.apache.hadoop.http.HttpServer2: Added filter
static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context static2016-05-25 23:09:49,563 INFO org.apache.hadoop.http.HttpServer2: Added filter
'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter)2016-05-25
23:09:49,566 INFO org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*2016-05-25 23:09:49,573 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to getDelegationToken2016-05-25 23:09:49,575 INFO org.apache.hadoop.http.HttpServer2:
Adding Kerberos (SPNEGO) filter to renewDelegationToken2016-05-25 23:09:49,577 INFO org.apache.hadoop.http.HttpServer2:
Adding Kerberos (SPNEGO) filter to cancelDelegationToken2016-05-25 23:09:49,579 INFO org.apache.hadoop.http.HttpServer2:
Adding Kerberos (SPNEGO) filter to fsck2016-05-25 23:09:49,581 INFO org.apache.hadoop.http.HttpServer2:
Adding Kerberos (SPNEGO) filter to imagetransfer2016-05-25 23:09:49,593 INFO org.apache.hadoop.http.HttpServer2:
Jetty bound to port 500702016-05-25 23:09:49,593 INFO org.mortbay.log: jetty-6.1.262016-05-25
23:09:49,954 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal HTTP/server1.cehdev.com@CEHDEV.COM2016-05-25
23:09:49,965 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: 'signature.secret'
configuration not set, using a random value as secret2016-05-25 23:09:49,967 INFO org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal HTTP/server1.cehdev.com@CEHDEV.COM2016-05-25
23:09:49,973 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: 'signature.secret'
configuration not set, using a random value as secret2016-05-25 23:09:50,037 INFO org.mortbay.log:
Started HttpServer2$SelectChannelConnectorWithSafeStartup@server1.cehdev.com:500702016-05-25
23:09:50,084 WARN org.apache.hadoop.hdfs.server.common.Util: Path /app/hadoop/dfs/name should
be specified as a URI in configuration files. Please update hdfs configuration.2016-05-25
23:09:50,085 WARN org.apache.hadoop.hdfs.server.common.Util: Path /app/hadoop/dfs/name should
be specified as a URI in configuration files. Please update hdfs configuration.2016-05-25
23:09:50,087 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack of redundant
storage directories!2016-05-25 23:09:50,095 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /app/hadoop/dfs/name should be specified as a URI in configuration files. Please update
hdfs configuration.2016-05-25 23:09:50,096 WARN org.apache.hadoop.hdfs.server.common.Util:
Path /app/hadoop/dfs/name should be specified as a URI in configuration files. Please update
hdfs configuration.2016-05-25 23:09:50,153 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsLock is fair:true2016-05-25 23:09:50,219 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=10002016-05-25 23:09:50,220 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.namenode.datanode.registration.ip-hostname-check=true2016-05-25 23:09:50,223 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.0002016-05-25 23:09:50,223
INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: The block deletion will start
around 2016 May 25 23:09:502016-05-25 23:09:50,226 INFO org.apache.hadoop.util.GSet: Computing
capacity for map BlocksMap2016-05-25 23:09:50,226 INFO org.apache.hadoop.util.GSet: VM type
      = 64-bit2016-05-25 23:09:50,228 INFO org.apache.hadoop.util.GSet: 2.0% max memory
889 MB = 17.8 MB2016-05-25 23:09:50,228 INFO org.apache.hadoop.util.GSet: capacity      =
2^21 = 2097152 entries2016-05-25 23:09:50,235 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=true2016-05-25 23:09:50,235 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.key.update.interval=600 min(s), dfs.block.access.token.lifetime=600 min(s),
dfs.encrypt.data.transfer.algorithm=rc42016-05-25 23:09:50,245 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
defaultReplication         = 22016-05-25 23:09:50,246 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplication             = 5122016-05-25 23:09:50,246 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
minReplication             = 12016-05-25 23:09:50,246 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams      = 22016-05-25 23:09:50,246 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks  = false2016-05-25 23:09:50,246 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 30002016-05-25 23:09:50,246 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
encryptDataTransfer        = true2016-05-25 23:09:50,246 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxNumBlocksToLog          = 10002016-05-25 23:09:50,247 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
fsOwner             = hdfs/server1.cehdev.com@CEHDEV.COM (auth:KERBEROS)2016-05-25 23:09:50,247
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          = hadoop2016-05-25
23:09:50,247 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
= true2016-05-25 23:09:50,248 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Determined
nameservice ID: hdcluster2016-05-25 23:09:50,248 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
HA Enabled: true2016-05-25 23:09:50,250 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Append Enabled: true2016-05-25 23:09:50,297 INFO org.apache.hadoop.util.GSet: Computing capacity
for map INodeMap2016-05-25 23:09:50,297 INFO org.apache.hadoop.util.GSet: VM type      
= 64-bit2016-05-25 23:09:50,298 INFO org.apache.hadoop.util.GSet: 1.0% max memory 889 MB =
8.9 MB2016-05-25 23:09:50,298 INFO org.apache.hadoop.util.GSet: capacity      = 2^20 =
1048576 entries2016-05-25 23:09:50,299 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
Caching file names occuring more than 10 times2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet:
Computing capacity for map cachedBlocks2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet:
VM type       = 64-bit2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet: 0.25% max
memory 889 MB = 2.2 MB2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet: capacity  
   = 2^18 = 262144 entries2016-05-25 23:09:50,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.99900001287460332016-05-25 23:09:50,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 02016-05-25 23:09:50,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension     = 300002016-05-25 23:09:50,313 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Retry cache on namenode is enabled2016-05-25 23:09:50,314 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis2016-05-25
23:09:50,317 INFO org.apache.hadoop.util.GSet: Computing capacity for map NameNodeRetryCache2016-05-25
23:09:50,317 INFO org.apache.hadoop.util.GSet: VM type       = 64-bit2016-05-25 23:09:50,317
INFO org.apache.hadoop.util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB2016-05-25
23:09:50,317 INFO org.apache.hadoop.util.GSet: capacity      = 2^15 = 32768 entries2016-05-25
23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf: ACLs enabled? false2016-05-25
23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf: XAttrs enabled? true2016-05-25
23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf: Maximum size of an xattr:
163842016-05-25 23:09:50,334 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /app/hadoop/dfs/name/in_use.lock
acquired by nodename 23849@server1.cehdev.com2016-05-25 23:09:51,198 WARN org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory:
The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded2016-05-25
23:09:51,785 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to
the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hdfs/server2.cehdev.com@CEHDEV.COM2016-05-25
23:09:51,786 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to
the server : java.lang.IllegalArgumentException: Server has invalid Kerberos principal: hdfs/server3.cehdev.com@CEHDEV.COM2016-05-25
23:09:51,924 WARN org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input
streams from QJM to [10.67.169.45:8485, 10.67.169.49:8485, 10.67.169.46:8485]. Skipping.org.apache.hadoop.hdfs.qjournal.client.QuorumException:
Got too many exceptions to achieve quorum size 2/3. 1 successful responses:10.67.169.45:8485:
[]2 exceptions thrown:10.67.169.46:8485: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException:
Server has invalid Kerberos principal: hdfs/server2.cehdev.com@CEHDEV.COM; Host Details :
local host is: "server1.cehdev.com/10.67.169.45"; destination host is: "server2.cehdev.com":8485;10.67.169.49:8485:
Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Server
has invalid Kerberos principal: hdfs/server3.cehdev.com@CEHDEV.COM; Host Details : local host
is: "server1.cehdev.com/10.67.169.45"; destination host is: "server3.cehdev.com":8485;   
    at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81) 
      at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223) 
      at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142) 
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:471) 
      at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:260) 
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1430) 
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1450) 
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:636) 
      at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279) 
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955) 
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700) 
      at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529) 
      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585) 
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:751) 
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:735) 
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407) 
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
Regards,Arul.

Mime
View raw message