nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pierre Villard (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (NIFI-4318) Processor cannot be stopped when Kerberos authentication default to prompt
Date Tue, 05 Sep 2017 20:12:00 GMT

     [ https://issues.apache.org/jira/browse/NIFI-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pierre Villard resolved NIFI-4318.
----------------------------------
    Resolution: Not A Bug

Closing as not a bug.

For the Hive processor, it's been solved using the validate query parameter in the controller
service.

For the HDFS processor, it was due to a relogin period time of 4 hours with a lifetime of
8 hours for generated tickets. The processor was not scheduled to run frequently and it was
possible to have an execution of the processor without renewing the ticket. This is due to
the unnecessary relogin period parameter that should be removed as it is already handled by
the hadoop library (renewal will happen at 80% of ticket lifetime). Will raise a specific
JIRA for that.

> Processor cannot be stopped when Kerberos authentication default to prompt
> --------------------------------------------------------------------------
>
>                 Key: NIFI-4318
>                 URL: https://issues.apache.org/jira/browse/NIFI-4318
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Extensions
>    Affects Versions: 1.3.0
>         Environment: 3-nodes cluster
>            Reporter: Pierre Villard
>         Attachments: image001.png, thread-2.txt, thread.txt
>
>
> I was unable to stop a PutHiveQL processor (it was showing running threads and remained
in this state at least half an hour). I had to restart NiFi to solve the situation. It looks
like the Kerberos authentication mechanism is falling back to manual user input and wait for
some input (see below promptForName):
> {noformat}
> "Timer-Driven Process Thread-2" Id=139 RUNNABLE  (in native code)
> 	at java.io.FileInputStream.readBytes(Native Method)
> 	at java.io.FileInputStream.read(FileInputStream.java:255)
> 	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> 	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> 	- waiting on java.io.BufferedInputStream@2e2d3f92
> 	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> 	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> 	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> 	- waiting on java.io.InputStreamReader@64628fdf
> 	at java.io.InputStreamReader.read(InputStreamReader.java:184)
> 	at java.io.BufferedReader.fill(BufferedReader.java:161)
> 	at java.io.BufferedReader.readLine(BufferedReader.java:324)
> 	- waiting on java.io.InputStreamReader@64628fdf
> 	at java.io.BufferedReader.readLine(BufferedReader.java:389)
> 	at com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
> 	at com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
> 	at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858)
> 	at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
> 	at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
> 	at sun.reflect.GeneratedMethodAccessor597.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
> 	at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
> 	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
> 	at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
> 	at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
> 	at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
> 	at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
> 	at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
> 	at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
> 	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
> 	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> 	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> 	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> 	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> 	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> 	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
> 	at org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> 	at org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
> 	at org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
> 	at org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
> 	at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
> 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
> 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
> 	at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
> 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
> 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
> 	at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
> 	at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
> 	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
> 	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:223)
> 	at org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:215)
> 	at sun.reflect.GeneratedMethodAccessor504.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1374)
> 	at com.sun.proxy.$Proxy361.ExecuteStatement(Unknown Source)
> 	at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:299)
> 	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
> 	at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)
> 	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
> 	at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
> 	at org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218)
> 	at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$507/743570245.apply(Unknown Source)
> 	at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
> 	at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199)
> 	at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$76/1354314579.apply(Unknown Source)
> 	at org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59)
> 	at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101)
> 	at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255)
> 	at org.apache.nifi.processors.hive.PutHiveQL$$Lambda$503/1913915475.execute(Unknown
Source)
> 	at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
> 	at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
> 	at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
> 	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
> 	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> 	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> 	at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> 	Number of Locked Synchronizers: 3
> 	- java.util.concurrent.locks.ReentrantLock$FairSync@179f2932
> 	- java.util.concurrent.locks.ReentrantLock$FairSync@5417b82e
> 	- java.util.concurrent.ThreadPoolExecutor$Worker@30eaacc3
> {noformat}
> I faced the same issue today with ListHDFS processor. Here is the extract from the thread
dump:
> {noformat}
> "Timer-Driven Process Thread-4" Id=160 RUNNABLE  (in native code)
>         at java.io.FileInputStream.readBytes(Native Method)
>         at java.io.FileInputStream.read(FileInputStream.java:255)
>         at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
>         at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
>         - waiting on java.io.BufferedInputStream@36e17d2a
>         at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
>         at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
>         at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
>         - waiting on java.io.InputStreamReader@3ae79dc6
>         at java.io.InputStreamReader.read(InputStreamReader.java:184)
>         at java.io.BufferedReader.fill(BufferedReader.java:161)
>         at java.io.BufferedReader.readLine(BufferedReader.java:324)
>         - waiting on java.io.InputStreamReader@3ae79dc6
>         at java.io.BufferedReader.readLine(BufferedReader.java:389)
>         at com.sun.security.auth.callback.TextCallbackHandler.readLine(TextCallbackHandler.java:153)
>         at com.sun.security.auth.callback.TextCallbackHandler.handle(TextCallbackHandler.java:120)
>         at com.sun.security.auth.module.Krb5LoginModule.promptForName(Krb5LoginModule.java:858)
>         at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:704)
>         at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
>         at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
>         at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
>         at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
>         at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
>         at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
>         at sun.security.jgss.GSSUtil.login(GSSUtil.java:258)
>         at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
>         at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
>         at sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
>         at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
>         at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
>         at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>         at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>         at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>         at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
>         at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
>         at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:560)
>         - waiting on org.apache.hadoop.ipc.Client$Connection@74a4e314
>         at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:375)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:729)
>         at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
>         at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
>         - waiting on org.apache.hadoop.ipc.Client$Connection@74a4e314
>         at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
>         at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1451)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>         at com.sun.proxy.$Proxy527.getListing(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
>         at sun.reflect.GeneratedMethodAccessor834.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy528.getListing(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2086)
>         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2069)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
>         at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
>         at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557)
>         at org.apache.nifi.processors.hadoop.ListHDFS.getStatuses(ListHDFS.java:388)
>         at org.apache.nifi.processors.hadoop.ListHDFS.onTrigger(ListHDFS.java:341)
>         at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>         at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>         at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>         at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:748)
>         Number of Locked Synchronizers: 1
>         - java.util.concurrent.ThreadPoolExecutor$Worker@69a9f59d
> {noformat}
> Since NiFi won't answer the prompt, it could be interesting to default doNotPrompt to
true so that authentication fails and can be retried ([source|http://docs.oracle.com/javase/7/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html]).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message