flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yoandy terradas <kw...@hotmail.com>
Subject issues authenticating on hdfs sink using kerberos
Date Wed, 13 Jul 2016 23:19:39 GMT

hi, 

im using flume 1.6.0 agent setup kafka-source, memory-channel and hdfs-sink.  the hdfs has
kerberos setup with proxy users.  i cannot get the sink to write because it fails saying i
didnt provide kerberos principal.  

2016-07-13 17:25:41,738 (conf-file-poller-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)]
Unable to load native-hadoop library for your platform... using builtin-java classes where
applicable
2016-07-13 17:25:41,853 (conf-file-poller-0) [INFO - org.apache.flume.auth.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:172)]
Attempting kerberos login as principal (ebee/01-ebee-autoenv.envnxs.net@REALM.COM) from keytab
file (/home/yt/flume/conf/krb5.keytab.ebee)
2016-07-13 17:25:42,252 (conf-file-poller-0) [INFO - org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:979)]
Login successful for user ebee/01-ebee-autoenv.envnxs.net@REALM.COM using keytab file /home/yt/flume/conf/krb5.keytab.ebee
2016-07-13 17:25:42,253 (conf-file-poller-0) [INFO - org.apache.flume.auth.KerberosAuthenticator.printUGI(KerberosAuthenticator.java:191)]

Logged as:  
User: ebee/01-ebee-autoenv.envnxs.net@REALM.COM
Auth method: KERBEROS 
Keytab: true 

2016-07-13 17:25:42,254 (conf-file-poller-0) [INFO - org.apache.flume.auth.KerberosAuthenticator.printUGI(KerberosAuthenticator.java:191)]

Proxy as:  
User: pub_user 
Auth method: PROXY 
Keytab: false 

2016-07-13 17:25:42,257 (conf-file-poller-0) [INFO - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:114)]
Channel mem-ch1 connected to [kafka-sc1, hdfs-sk1]
 
that looks like authentication was successful.  it connects to kafka and start putting messages
in the channel, then 

2016-07-13 17:25:44,304 (ConsumerFetcherThread-flume_01.local-hostname-1468430742665-7ece6605-0-80)
[INFO - kafka.utils.Logging$class.info(Logging.scala:68)] [ConsumerFetcherThread-flume_01.rufus.sand-08.lax1-1468430742665-7ece6605-0-80],
Starting 
2016-07-13 17:25:44,643 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.HDFSSequenceFile.configure(HDFSSequenceFile.java:63)]
writeFormat = Text, UseRawLocalFileSystem = false
2016-07-13 17:25:44,695 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:234)]
Creating hdfs://01-hadoop-namenode-hostname/pubs/logs/opt_rtpp_mapper_log_record_joined_pb/16-07-13-17-25/events.1468430744636.tmp
2016-07-13 17:25:45,918 (hdfs-hdfs-sk1-call-runner-0) [WARN - org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:675)]
Exception encountered while connecting to the server : java.lang.IllegalArgumentException:
Failed to specify server's Kerberos principal name
2016-07-13 17:25:45,919 (hdfs-hdfs-sk1-call-runner-0) [WARN - org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1674)]
PriviledgedActionException as:ebee/01-ebee-autoenv.envnxs.net@REALM.COM (auth:KERBEROS) cause:java.io.IOException:
java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
2016-07-13 17:25:45,933 (hdfs-hdfs-sk1-call-runner-0) [WARN - org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1674)]
PriviledgedActionException as:dpaas_publishers_user (auth:PROXY) via ebee/01-ebee-autoenv.envnxs.net@REAL.COM
(auth:KERBEROS) cause:java.io.IOException: Failed on local exception: java.io.IOException:
java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host
Details : local host is: "01.local-hostname/local-ip-address"; destination host is: "01-hadoop-namenode-hostname":8020;

2016-07-13 17:25:45,935 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)]
process failed
org.apache.flume.auth.SecurityException: Privileged action failed
at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49)
at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException:
Failed to specify server's Kerberos principal name; Host Details : local host is: "01.local-hostname/local-ip-address";
destination host is: "01-hadoop-namenode-hostname":8020; 
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

looking at the code for hdfs-sink it seems as the kerberos auth is propagated.  any idea as
to why it seems to have authenticated but it didnt? 

/yoandy
Mime
View raw message