hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gagan Brahmi <gaganbra...@gmail.com>
Subject Re: Hadoop Security - NN fails to connect to JN on different servers
Date Fri, 27 May 2016 16:19:20 GMT
One of the potential problem might be with the hostname
configurations. If you are using the host file to resolve the DNS
please verify the hostnames set in the host file. The problem looks to
be related with the invalid principal name which can be due to bad
hostname to IP mapping.


Regards,
Gagan Brahmi

On Wed, May 25, 2016 at 11:11 AM, Arulanand Dayalan
<arul_anand_2000@yahoo.com.invalid> wrote:
> Hi,
>
> I had setup Hadoop HA Cluster(QJM) with kerberos. Cluster has three journal
> nodes. Namenode is able to connect
> to journal node running on the Namenode. For other Journal nodes its not
> able to connect and throws below exception.
> Can you please provide inputs for resolving the issue.
>
> 2016-05-25 23:09:47,526 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = server1.cehdev.com/10.67.169.45
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 2.5.2
> STARTUP_MSG:   classpath =
> STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r
> cc72e9b000545b86b75a61f4835eb86d57bfafc0; compiled by 'jenkins' on
> 2014-11-14T23:45Z
> STARTUP_MSG:   java = 1.7.0_79
> ************************************************************/
> 2016-05-25 23:09:47,550 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal
> handlers for [TERM, HUP, INT]
> 2016-05-25 23:09:47,557 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
> 2016-05-25 23:09:48,147 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
> loaded properties from hadoop-metrics2.properties
> 2016-05-25 23:09:48,307 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> at 10 second(s).
> 2016-05-25 23:09:48,307 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2016-05-25 23:09:48,309 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is
> hdfs://hdcluster
> 2016-05-25 23:09:48,310 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use
> hdcluster to access this namenode/service.
> 2016-05-25 23:09:49,331 INFO
> org.apache.hadoop.security.UserGroupInformation: Login successful for user
> hdfs/server1.cehdev.com@CEHDEV.COM using keytab file
> /etc/security/keytab/hdfs.service.keytab
> 2016-05-25 23:09:49,392 INFO org.apache.hadoop.hdfs.DFSUtil: Starting web
> server as: HTTP/server1.cehdev.com@CEHDEV.COM
> 2016-05-25 23:09:49,392 INFO org.apache.hadoop.hdfs.DFSUtil: Starting
> Web-server for hdfs at: http://server1.cehdev.com:50070
> 2016-05-25 23:09:49,480 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2016-05-25 23:09:49,487 INFO org.apache.hadoop.http.HttpRequestLog: Http
> request log for http.requests.namenode is not defined
> 2016-05-25 23:09:49,504 INFO org.apache.hadoop.http.HttpServer2: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2016-05-25 23:09:49,510 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context hdfs
> 2016-05-25 23:09:49,510 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2016-05-25 23:09:49,511 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2016-05-25 23:09:49,563 INFO org.apache.hadoop.http.HttpServer2: Added
> filter 'org.apache.hadoop.hdfs.web.AuthFilter'
> (class=org.apache.hadoop.hdfs.web.AuthFilter)
> 2016-05-25 23:09:49,566 INFO org.apache.hadoop.http.HttpServer2:
> addJerseyResourcePackage:
> packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
> pathSpec=/webhdfs/v1/*
> 2016-05-25 23:09:49,573 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to getDelegationToken
> 2016-05-25 23:09:49,575 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to renewDelegationToken
> 2016-05-25 23:09:49,577 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to cancelDelegationToken
> 2016-05-25 23:09:49,579 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to fsck
> 2016-05-25 23:09:49,581 INFO org.apache.hadoop.http.HttpServer2: Adding
> Kerberos (SPNEGO) filter to imagetransfer
> 2016-05-25 23:09:49,593 INFO org.apache.hadoop.http.HttpServer2: Jetty bound
> to port 50070
> 2016-05-25 23:09:49,593 INFO org.mortbay.log: jetty-6.1.26
> 2016-05-25 23:09:49,954 INFO
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
> Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal
> HTTP/server1.cehdev.com@CEHDEV.COM
> 2016-05-25 23:09:49,965 WARN
> org.apache.hadoop.security.authentication.server.AuthenticationFilter:
> 'signature.secret' configuration not set, using a random value as secret
> 2016-05-25 23:09:49,967 INFO
> org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
> Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal
> HTTP/server1.cehdev.com@CEHDEV.COM
> 2016-05-25 23:09:49,973 WARN
> org.apache.hadoop.security.authentication.server.AuthenticationFilter:
> 'signature.secret' configuration not set, using a random value as secret
> 2016-05-25 23:09:50,037 INFO org.mortbay.log: Started
> HttpServer2$SelectChannelConnectorWithSafeStartup@server1.cehdev.com:50070
> 2016-05-25 23:09:50,084 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /app/hadoop/dfs/name should be specified as a URI in configuration files.
> Please update hdfs configuration.
> 2016-05-25 23:09:50,085 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /app/hadoop/dfs/name should be specified as a URI in configuration files.
> Please update hdfs configuration.
> 2016-05-25 23:09:50,087 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of data loss due to
> lack of redundant storage directories!
> 2016-05-25 23:09:50,095 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /app/hadoop/dfs/name should be specified as a URI in configuration files.
> Please update hdfs configuration.
> 2016-05-25 23:09:50,096 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /app/hadoop/dfs/name should be specified as a URI in configuration files.
> Please update hdfs configuration.
> 2016-05-25 23:09:50,153 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is fair:true
> 2016-05-25 23:09:50,219 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2016-05-25 23:09:50,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.namenode.datanode.registration.ip-hostname-check=true
> 2016-05-25 23:09:50,223 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
> 2016-05-25 23:09:50,223 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: The block
> deletion will start around 2016 May 25 23:09:50
> 2016-05-25 23:09:50,226 INFO org.apache.hadoop.util.GSet: Computing capacity
> for map BlocksMap
> 2016-05-25 23:09:50,226 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2016-05-25 23:09:50,228 INFO org.apache.hadoop.util.GSet: 2.0% max memory
> 889 MB = 17.8 MB
> 2016-05-25 23:09:50,228 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^21 = 2097152 entries
> 2016-05-25 23:09:50,235 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=true
> 2016-05-25 23:09:50,235 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.key.update.interval=600 min(s),
> dfs.block.access.token.lifetime=600 min(s),
> dfs.encrypt.data.transfer.algorithm=rc4
> 2016-05-25 23:09:50,245 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication         = 2
> 2016-05-25 23:09:50,246 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2016-05-25 23:09:50,246 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2016-05-25 23:09:50,246 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams      = 2
> 2016-05-25 23:09:50,246 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks  = false
> 2016-05-25 23:09:50,246 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2016-05-25 23:09:50,246 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer        = true
> 2016-05-25 23:09:50,246 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxNumBlocksToLog          = 1000
> 2016-05-25 23:09:50,247 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             =
> hdfs/server1.cehdev.com@CEHDEV.COM (auth:KERBEROS)
> 2016-05-25 23:09:50,247 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          =
> hadoop
> 2016-05-25 23:09:50,247 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2016-05-25 23:09:50,248 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Determined nameservice
> ID: hdcluster
> 2016-05-25 23:09:50,248 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: true
> 2016-05-25 23:09:50,250 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2016-05-25 23:09:50,297 INFO org.apache.hadoop.util.GSet: Computing capacity
> for map INodeMap
> 2016-05-25 23:09:50,297 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2016-05-25 23:09:50,298 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> 889 MB = 8.9 MB
> 2016-05-25 23:09:50,298 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^20 = 1048576 entries
> 2016-05-25 23:09:50,299 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
> more than 10 times
> 2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet: Computing capacity
> for map cachedBlocks
> 2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet: 0.25% max memory
> 889 MB = 2.2 MB
> 2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^18 = 262144 entries
> 2016-05-25 23:09:50,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2016-05-25 23:09:50,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2016-05-25 23:09:50,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension     = 30000
> 2016-05-25 23:09:50,313 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode
> is enabled
> 2016-05-25 23:09:50,314 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
> 2016-05-25 23:09:50,317 INFO org.apache.hadoop.util.GSet: Computing capacity
> for map NameNodeRetryCache
> 2016-05-25 23:09:50,317 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2016-05-25 23:09:50,317 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory 889 MB = 273.1 KB
> 2016-05-25 23:09:50,317 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^15 = 32768 entries
> 2016-05-25 23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf:
> ACLs enabled? false
> 2016-05-25 23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf:
> XAttrs enabled? true
> 2016-05-25 23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf:
> Maximum size of an xattr: 16384
> 2016-05-25 23:09:50,334 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /app/hadoop/dfs/name/in_use.lock acquired by nodename
> 23849@server1.cehdev.com
> 2016-05-25 23:09:51,198 WARN
> org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 2016-05-25 23:09:51,785 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> java.lang.IllegalArgumentException: Server has invalid Kerberos principal:
> hdfs/server2.cehdev.com@CEHDEV.COM
> 2016-05-25 23:09:51,786 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server :
> java.lang.IllegalArgumentException: Server has invalid Kerberos principal:
> hdfs/server3.cehdev.com@CEHDEV.COM
> 2016-05-25 23:09:51,924 WARN
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input
> streams from QJM to [10.67.169.45:8485, 10.67.169.49:8485,
> 10.67.169.46:8485]. Skipping.
> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
> exceptions to achieve quorum size 2/3. 1 successful responses:
> 10.67.169.45:8485: []
> 2 exceptions thrown:
> 10.67.169.46:8485: Failed on local exception: java.io.IOException:
> java.lang.IllegalArgumentException: Server has invalid Kerberos principal:
> hdfs/server2.cehdev.com@CEHDEV.COM; Host Details : local host is:
> "server1.cehdev.com/10.67.169.45"; destination host is:
> "server2.cehdev.com":8485;
> 10.67.169.49:8485: Failed on local exception: java.io.IOException:
> java.lang.IllegalArgumentException: Server has invalid Kerberos principal:
> hdfs/server3.cehdev.com@CEHDEV.COM; Host Details : local host is:
> "server1.cehdev.com/10.67.169.45"; destination host is:
> "server3.cehdev.com":8485;
>         at
> org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
>         at
> org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
>         at
> org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
>         at
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:471)
>         at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:260)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1430)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1450)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:636)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:751)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:735)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
>
> Regards,
> Arul.
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org


Mime
View raw message