hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elazar Leibovich <elaz...@gmail.com>
Subject Re: "Server not found in Kerberos database" for MiniKDC server
Date Mon, 20 Jan 2014 11:34:13 GMT
For the sake of completion.

The same settings worked in a Linux box.


On Wed, Jan 15, 2014 at 10:57 PM, Elazar Leibovich <elazarl@gmail.com>wrote:

> Hi,
>
> For educational purposes, I'm trying to set a minimal working secure
> Hadoop cluster on my machine.
>
> What I basically did is:
>
> Add example.com to /etc/hosts
>
> Set a minkdc server. It'll generate krb5.conf and keytab. Generates some
> users - {nn,dn,hdfs}@EXAMPLE.COM
>
> Refer Java to krb5.conf with HADOOP_OPTS, as well as a required workaround
> for Mac OS X java:
>
> ❯ ~/hadoopconf env HADOOP_OPTS
> hadoop-env.sh HADOOP_OPTS = -Djava.awt.headless=true
> -Djava.security.krb5.conf=/Users/eleibovi/dev/securehadoop/hadoop_rpc_walktrhough/krb5.conf
> -Djava.net.preferIPv4Stack=true
>
>
> Set the proper hadoop configuration using the keytab and the hadoop users:
>
> ❯ ~/hadoopconf get --local
> hdfs-site.xml dfs.datanode.address            = example.com:1004
> core-site.xml fs.defaultFS                    = hdfs://example.com
> hdfs-site.xml dfs.namenode.keytab.file        =
> /Users/eleibovi/dev/securehadoop/hadoop_rpc_walktrhough/keytab
> hdfs-site.xml dfs.datanode.hostname           = example.com
> hdfs-site.xml dfs.datanode.kerberos.principal = dn/EXAMPLE.COM@EXAMPLE.COM
> hdfs-site.xml dfs.datanode.data.dir           =
> /tmp/hadoop-eleibovi/dfs/data
> hdfs-site.xml dfs.datanode.keytab.file        =
> /Users/eleibovi/dev/securehadoop/hadoop_rpc_walktrhough/keytab
> hdfs-site.xml dfs.namenode.kerberos.principal = nn/EXAMPLE.COM@EXAMPLE.COM
> core-site.xml hadoop.security.authorization   = true
> core-site.xml hadoop.security.authentication  = kerberos
> hdfs-site.xml dfs.datanode.dns.interface      = lo0
> hdfs-site.xml dfs.datanode.http.address       = example.com:1006
>
> Start the namenode service.
> $ ./bin/hdfs
> ...
> 14/01/15 19:22:43 INFO ipc.Server: IPC Server listener on 8020: starting
> 14/01/15 19:22:43 INFO namenode.NameNode: NameNode RPC up at: localhost/
> 127.0.0.1:8020
> 14/01/15 19:22:43 INFO namenode.FSNamesystem: Starting services required
> for active state
>
> Finally use the following short Java program to contact the namenode:
>
> System.setProperty("java.security.krb5.conf", cwd + "/krb5.conf");
> UserGroupInformation.setConfiguration(conf);
>         UserGroupInformation ugi = UserGroupInformation.
>                 loginUserFromKeytabAndReturnUGI("hdfs/EXAMPLE.COM", cwd +
> "/keytab");
>  ugi.doAs(new PrivilegedExceptionAction<Object>() {
>             @Override
>             public Object run() throws Exception {
>                 final FileSystem fs = FileSystem.get(conf);
>                 fs.getFileStatus(new Path("/"));
>          }
> }
>
> The exception I got is:
>
> Exception in thread "main" java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism level:
> Server not found in Kerberos database (7) - Server not found in Kerberos
> database)]; Host Details : local host is: "tlv-mpbxb/127.0.0.1";
> destination host is: "example.com":8020;
>
> I'll be glad to any help with debugging the problem.
>
> Thanks,
>
> I attach a full log with Kerberos debug turned on:
>
> args: [-conf,
> /Users/eleibovi/dev/securehadoop/hadoop-2.1.0-beta/etc/hadoop/core-site.xml,
> -conf,
> /Users/eleibovi/dev/securehadoop/hadoop-2.1.0-beta/etc/hadoop/hdfs-site.xml]
> 2014-01-15 19:29:46 DEBUG MutableMetricsFactory:42 - field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
> with annotation
> @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=,
> value=[Rate of successful kerberos logins and latency (milliseconds)],
> always=false, type=DEFAULT, sampleName=Ops)
> 2014-01-15 19:29:46 DEBUG MutableMetricsFactory:42 - field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
> with annotation
> @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time, about=,
> value=[Rate of failed kerberos logins and latency (milliseconds)],
> always=false, type=DEFAULT, sampleName=Ops)
> 2014-01-15 19:29:46 DEBUG MetricsSystemImpl:220 - UgiMetrics, User and
> group related metrics
> Java config name:
> /Users/eleibovi/dev/securehadoop/hadoop_rpc_walktrhough/krb5.conf
> Loaded from Java config
> 2014-01-15 19:29:46 DEBUG Groups:180 -  Creating new Groups object
> 2014-01-15 19:29:46 DEBUG NativeCodeLoader:46 - Trying to load the
> custom-built native-hadoop library...
> 2014-01-15 19:29:46 DEBUG NativeCodeLoader:55 - Failed to load
> native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in
> java.library.path
> 2014-01-15 19:29:46 DEBUG NativeCodeLoader:56 -
> java.library.path=/Users/eleibovi/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
> 2014-01-15 19:29:46 WARN  NativeCodeLoader:62 - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2014-01-15 19:29:46 DEBUG JniBasedUnixGroupsMappingWithFallback:40 -
> Falling back to shell based
> 2014-01-15 19:29:46 DEBUG JniBasedUnixGroupsMappingWithFallback:44 - Group
> mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
> 2014-01-15 19:29:46 DEBUG Groups:66 - Group mapping
> impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback;
> cacheTimeout=300000
> Java config name:
> /Users/eleibovi/dev/securehadoop/hadoop_rpc_walktrhough/krb5.conf
> Loaded from Java config
> >>> KdcAccessibility: reset
> >>> KdcAccessibility: reset
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): nn
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 53; type: 3
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): nn
>  >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 69; type: 16
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): nn
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 61; type: 17
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): nn
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 61; type: 23
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): dn
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 53; type: 3
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): dn
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 69; type: 16
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): dn
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 61; type: 17
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): dn
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 61; type: 23
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): eleibovi
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 59; type: 3
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): eleibovi
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 75; type: 16
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): eleibovi
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 67; type: 17
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): eleibovi
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 67; type: 23
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 55; type: 3
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 71; type: 16
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 63; type: 17
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTab: load() entry length: 63; type: 23
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTab: load() entry length: 42; type: 3
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTab: load() entry length: 58; type: 16
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTab: load() entry length: 50; type: 17
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): hdfs
> >>> KeyTab: load() entry length: 50; type: 23
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): root
> >>> KeyTab: load() entry length: 42; type: 3
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): root
> >>> KeyTab: load() entry length: 58; type: 16
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): root
> >>> KeyTab: load() entry length: 50; type: 17
> >>> KeyTabInputStream, readName(): EXAMPLE.COM
> >>> KeyTabInputStream, readName(): root
> >>> KeyTab: load() entry length: 50; type: 23
> Added key: 23version: 0
> Added key: 17version: 0
> Added key: 16version: 0
> Added key: 3version: 0
> Ordering keys wrt default_tkt_enctypes list
> Using builtin default etypes for default_tkt_enctypes
> default etypes for default_tkt_enctypes: 18 17 16 23 1 3.
> Added key: 23version: 0
> Added key: 17version: 0
> Added key: 16version: 0
> Added key: 3version: 0
> Ordering keys wrt default_tkt_enctypes list
> Using builtin default etypes for default_tkt_enctypes
> default etypes for default_tkt_enctypes: 18 17 16 23 1 3.
> Using builtin default etypes for default_tkt_enctypes
> default etypes for default_tkt_enctypes: 18 17 16 23 1 3.
> >>> KrbAsReq creating message
> >>> KrbKdcReq send: kdc=localhost TCP:50064, timeout=30000, number of
> retries =3, #bytes=158
> >>> KDCCommunication: kdc=localhost TCP:50064, timeout=30000,Attempt =1,
> #bytes=158
> >>>DEBUG: TCPClient reading 529 bytes
> >>> KrbKdcReq send: #bytes read=529
> >>> KdcAccessibility: remove localhost:50064
> Added key: 23version: 0
> Added key: 17version: 0
> Added key: 16version: 0
> Added key: 3version: 0
> Ordering keys wrt default_tkt_enctypes list
> Using builtin default etypes for default_tkt_enctypes
> default etypes for default_tkt_enctypes: 18 17 16 23 1 3.
> >>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
> >>> KrbAsRep cons in KrbAsReq.getReply hdfs/EXAMPLE.COM
> 2014-01-15 19:29:47 DEBUG UserGroupInformation:176 - hadoop login
> Added key: 23version: 0
> Added key: 17version: 0
> Added key: 16version: 0
> Added key: 3version: 0
> Ordering keys wrt default_tkt_enctypes list
> Using builtin default etypes for default_tkt_enctypes
> default etypes for default_tkt_enctypes: 18 17 16 23 1 3.
> 2014-01-15 19:29:47 DEBUG UserGroupInformation:125 - hadoop login commit
> 2014-01-15 19:29:47 DEBUG UserGroupInformation:139 - using kerberos
> user:hdfs/EXAMPLE.COM@EXAMPLE.COM
> 2014-01-15 19:29:47 DEBUG UserGroupInformation:1499 - PrivilegedAction
> as:hdfs/EXAMPLE.COM@EXAMPLE.COM (auth:KERBEROS)
> from:com.github.elazar.hadoop.examples.HadoopBasicUsage.run(HadoopBasicUsage.java:34)
> >>KERBEROS
> 2014-01-15 19:29:47 DEBUG BlockReaderLocal:326 -
> dfs.client.use.legacy.blockreader.local = false
> 2014-01-15 19:29:47 DEBUG BlockReaderLocal:329 -
> dfs.client.read.shortcircuit = false
> 2014-01-15 19:29:47 DEBUG BlockReaderLocal:332 -
> dfs.client.domain.socket.data.traffic = false
> 2014-01-15 19:29:47 DEBUG BlockReaderLocal:335 - dfs.domain.socket.path =
> 2014-01-15 19:29:47 DEBUG MetricsSystemImpl:220 - StartupProgress,
> NameNode startup progress
> 2014-01-15 19:29:47 DEBUG RetryUtils:74 - multipleLinearRandomRetry = null
> 2014-01-15 19:29:47 DEBUG Server:220 - rpcKind=RPC_PROTOCOL_BUFFER,
> rpcRequestWrapperClass=class
> org.apache.hadoop.ipc.ProtobufRpcEngine$RpcRequestWrapper,
> rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@379b0d9c
> 2014-01-15 19:29:47 DEBUG BlockReaderLocal:63 - Both short-circuit local
> reads and UNIX domain socket are disabled.
> 2014-01-15 19:29:47 DEBUG Shell:237 - Failed to detect a valid hadoop home
> directory
> java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
> at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:219)
> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:244)
>  at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
> at
> org.apache.hadoop.conf.Configuration.getTrimmedStrings(Configuration.java:1539)
>  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:492)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:445)
>  at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2429)
>  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2463)
>  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2445)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:363)
>  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:165)
> at
> com.github.elazar.hadoop.examples.HadoopBasicUsage$1.run(HadoopBasicUsage.java:38)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at
> com.github.elazar.hadoop.examples.HadoopBasicUsage.run(HadoopBasicUsage.java:34)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>  at
> com.github.elazar.hadoop.examples.HadoopBasicUsage.main(HadoopBasicUsage.java:18)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
> 2014-01-15 19:29:47 DEBUG Shell:316 - setsid is not available on this
> machine. So not using it.
> 2014-01-15 19:29:47 DEBUG Shell:320 - setsid exited with exit code 0
> 2014-01-15 19:29:47 DEBUG Client:371 - The ping interval is 60000 ms.
> 2014-01-15 19:29:47 DEBUG Client:636 - Connecting to
> example.com/127.0.0.1:8020
> 2014-01-15 19:29:47 DEBUG UserGroupInformation:1499 - PrivilegedAction
> as:hdfs/EXAMPLE.COM@EXAMPLE.COM (auth:KERBEROS)
> from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:654)
> 2014-01-15 19:29:47 DEBUG SaslRpcClient:438 - Sending sasl message state:
> NEGOTIATE
>
> 2014-01-15 19:29:47 DEBUG SaslRpcClient:370 - Received SASL message state:
> NEGOTIATE
> auths {
>   method: "TOKEN"
>   mechanism: "DIGEST-MD5"
>   protocol: ""
>   serverId: "default"
>   challenge:
> "realm=\"default\",nonce=\"Evyt9cWyZFDUbCtQYNcgF5FY7rsBqxVCgtggY48n\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
> }
> auths {
>   method: "KERBEROS"
>   mechanism: "GSSAPI"
>   protocol: "nn"
>   serverId: "EXAMPLE.COM"
> }
>
> 2014-01-15 19:29:47 DEBUG SaslRpcClient:259 - Get token info
> proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
> info:@org.apache.hadoop.security.token.TokenInfo(value=class
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)
> 2014-01-15 19:29:47 DEBUG SaslRpcClient:287 - Get kerberos info
> proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
> info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=,
> serverPrincipal=dfs.namenode.kerberos.principal)
> 2014-01-15 19:29:47 DEBUG SaslRpcClient:231 - RPC Server's Kerberos
> principal name for
> protocol=org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB is nn/
> EXAMPLE.COM@EXAMPLE.COM
> 2014-01-15 19:29:47 DEBUG SaslRpcClient:242 - Creating SASL
> GSSAPI(KERBEROS)  client to authenticate to service at EXAMPLE.COM
> 2014-01-15 19:29:47 DEBUG SaslRpcClient:172 - Use KERBEROS authentication
> for protocol ClientNamenodeProtocolPB
> Found ticket for hdfs/EXAMPLE.COM@EXAMPLE.COM to go to krbtgt/
> EXAMPLE.COM@EXAMPLE.COM expiring on Thu Jan 16 19:29:46 IST 2014
> Entered Krb5Context.initSecContext with state=STATE_NEW
> Found ticket for hdfs/EXAMPLE.COM@EXAMPLE.COM to go to krbtgt/
> EXAMPLE.COM@EXAMPLE.COM expiring on Thu Jan 16 19:29:46 IST 2014
> Service ticket not found in the subject
> >>> Credentials acquireServiceCreds: same realm
> Using builtin default etypes for default_tgs_enctypes
> default etypes for default_tgs_enctypes: 18 17 16 23 1 3.
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
> >>> EType: sun.security.krb5.internal.crypto.Aes128CtsHmacSha1EType
> >>> KrbKdcReq send: kdc=localhost TCP:50064, timeout=30000, number of
> retries =3, #bytes=583
> >>> KDCCommunication: kdc=localhost TCP:50064, timeout=30000,Attempt =1,
> #bytes=583
> >>>DEBUG: TCPClient reading 135 bytes
> >>> KrbKdcReq send: #bytes read=135
> >>> KdcAccessibility: remove localhost:50064
> >>> KDCRep: init() encoding tag is 126 req type is 13
> >>>KRBError:
>  sTime is Wed Jan 15 19:29:47 IST 2014 1389806987000
>  suSec is 0
>  error code is 7
>  error Message is Server not found in Kerberos database
>  realm is EXAMPLE.COM
>  sname is krbtgt/EXAMPLE.COM
>  msgType is 30
> 2014-01-15 19:29:47 ERROR UserGroupInformation:1480 -
> PriviledgedActionException as:hdfs/EXAMPLE.COM@EXAMPLE.COM(auth:KERBEROS) cause:javax.security.sasl.SaslException:
GSS initiate
> failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Server not found in Kerberos database (7) - Server not found in
> Kerberos database)]
> 2014-01-15 19:29:47 DEBUG UserGroupInformation:1499 - PrivilegedAction
> as:hdfs/EXAMPLE.COM@EXAMPLE.COM (auth:KERBEROS)
> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:583)
> >>>KinitOptions cache name is /Users/eleibovi/krb5cc_eleibovi
> >> Acquire default native Credentials
> >>> Obtained TGT from LSA: Credentials:
> *...Here it looks like it tries to get the machines kerberos tickets...*
> client=eleibovi@[snipped computer Kerberos server from /etc/krb5.conf]
> server=krbtgt/[snipped computer Kerberos server from /etc/krb5.conf]
> authTime=20140109035156Z
> startTime=20140109115308Z
> endTime=20140109215308Z
> renewTill=20140116035156Z
> flags: FORWARDABLE;RENEWABLE;INITIAL;PRE-AUTHENT
> EType (int): 18
> Using builtin default etypes for default_tgs_enctypes
> default etypes for default_tgs_enctypes: 18 17 16 23 1 3.
> >>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
> >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
> getKDCFromDNS using UDP
> getKDCFromDNS using TCP
> 2014-01-15 19:30:18 DEBUG UserGroupInformation:176 - hadoop login
> 2014-01-15 19:30:18 DEBUG UserGroupInformation:125 - hadoop login commit
> 2014-01-15 19:30:18 DEBUG UserGroupInformation:139 - using kerberos
> user:null
> 2014-01-15 19:30:18 DEBUG UserGroupInformation:155 - using local
> user:UnixPrincipal: eleibovi
> 2014-01-15 19:30:18 DEBUG UserGroupInformation:696 - UGI
> loginUser:eleibovi (auth:KERBEROS)
> 2014-01-15 19:30:18 WARN  Client:615 - Exception encountered while
> connecting to the server : javax.security.sasl.SaslException: GSS initiate
> failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Server not found in Kerberos database (7) - Server not found in
> Kerberos database)]
> 2014-01-15 19:30:18 ERROR UserGroupInformation:1480 -
> PriviledgedActionException as:hdfs/EXAMPLE.COM@EXAMPLE.COM(auth:KERBEROS) cause:java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - Server not found in Kerberos database)]
> 2014-01-15 19:30:18 DEBUG Client:1099 - closing ipc connection to
> example.com/127.0.0.1:8020: javax.security.sasl.SaslException: GSS
> initiate failed [Caused by GSSException: No valid credentials provided
> (Mechanism level: Server not found in Kerberos database (7) - Server not
> found in Kerberos database)]
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate
> failed [Caused by GSSException: No valid credentials provided (Mechanism
> level: Server not found in Kerberos database (7) - Server not found in
> Kerberos database)]
>  at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:620)
> at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
>  at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:583)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:667)
>  at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
>  at org.apache.hadoop.ipc.Client.call(Client.java:1318)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
>  at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
>  at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:188)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>  at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:651)
>  at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1636)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1117)
>  at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1113)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
>  at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1113)
> at
> com.github.elazar.hadoop.examples.HadoopBasicUsage$1.run(HadoopBasicUsage.java:40)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at
> com.github.elazar.hadoop.examples.HadoopBasicUsage.run(HadoopBasicUsage.java:34)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>  at
> com.github.elazar.hadoop.examples.HadoopBasicUsage.main(HadoopBasicUsage.java:18)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused
> by GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - Server not found in Kerberos database)]
>  at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:394)
>  at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:494)
> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:314)
>  at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:659)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:655)
>  at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:654)
>  ... 32 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Server not found in Kerberos database (7) - Server not found in Kerberos
> database)
>  at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
>  at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
>  ... 41 more
> Caused by: KrbException: Server not found in Kerberos database (7) -
> Server not found in Kerberos database
> at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
>  at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:192)
> at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:203)
>  at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:311)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:115)
>  at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:449)
> at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
>  ... 44 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at sun.security.krb5.internal.KDCRep.init(KDCRep.java:143)
>  at sun.security.krb5.internal.TGSRep.init(TGSRep.java:66)
> at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:61)
>  at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
> ... 50 more
> 2014-01-15 19:30:18 DEBUG Client:1107 - IPC Client (23781497) connection
> to example.com/127.0.0.1:8020 from hdfs/EXAMPLE.COM@EXAMPLE.COM: closed
>

Mime
View raw message