nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Georg Heiler <georg.kf.hei...@gmail.com>
Subject Re: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew tickets
Date Thu, 11 Jan 2018 17:50:12 GMT
Sure, here is the result (as an attachment)
looks like a problem regarding the locale for timestamps


Joe Witt <joe.witt@gmail.com> schrieb am Do., 11. Jan. 2018 um 10:01 Uhr:

> Please do share the test failure in poi.  The build is stable in Travis
> and most of us build in OSX all the time so basic build tests should be
> pretty stable and we always want that to be the case.
>  You can also just run with skip tests.
>
> mvn clean install -DskipTests -Pdir-only
>
> In NiFi 1.5.0 changes were made to Hive, HDFS, HBase code regarding
> handling of Kerberos tickets.
> https://issues.apache.org/jira/browse/NIFI-3472
>
> It is very possible this improves your situation.  It is best we do not
> get into vendor specific support here but rather we can speak to the Apache
> NiFi code this community puts out.
>
> Thanks
>
> On Thu, Jan 11, 2018 at 1:48 AM, Georg Heiler <georg.kf.heiler@gmail.com>
> wrote:
>
>> By now I can confirm that according to the documentation it must be the
>> old NiFi 1.2 according to our admin.
>>
>> Also, following your suggestion to try 1.5  fails as
>> [INFO] nifi-poi-processors ................................ FAILURE [
>>  6.617 s]
>> to build on OsX.
>>
>> @Juan Pablo Gardella:
>> Client {
>>   com.sun.security.auth.module.Krb5LoginModule required
>>   useKeyTab=true
>>   keyTab="xxx""
>>   principal="dddd"
>>   storeKey=true
>> *  useTicketCache=false*
>>   serviceName="zookeeper";
>> };
>> should be located in nifi_jaas.conf ?
>> Anyway, useTicketCache is already set to false.
>>
>>
>>
>> Schneider, Jonathan <jon@sclhs.net> schrieb am Mi., 10. Jan. 2018 um
>> 17:26 Uhr:
>>
>>> I don’t have the ability to test 1.5.0 RC at this time.  I’m not sure
>>> how well you could over-lay it over HDF’s flavor of Nifi.  If I get a
>>> chance, I’ll try to load a new server and load 1.5 on it, then load my
>>> flows into it.  The problem I can see having is regarding joining the
>>> Ambari managed Kerberos realm.
>>>
>>>
>>>
>>> *Jonathan Schneider*
>>>
>>> Hadoop/UNIX Administrator, STSC
>>>
>>> SCL Health
>>>
>>> 17501 W. 98
>>> <https://maps.google.com/?q=17501+W.+98&entry=gmail&source=g>th
St,
>>> Pillars 25-33
>>>
>>> Lenexa, KS  66219
>>>
>>> P: 913.895.2999 <(913)%20895-2999>
>>>
>>> jon@sclhs.net <jon@sclhs.net%20%0d>
>>>
>>> www.sclhealthsystem.org
>>>
>>>
>>>
>>> [image: image001.jpg]
>>>
>>>
>>>
>>> *From:* Matt Burgess [mailto:mattyb149@apache.org]
>>> *Sent:* Wednesday, January 10, 2018 10:24 AM
>>>
>>>
>>> *To:* users@nifi.apache.org
>>> *Subject:* Re: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew
>>> tickets
>>>
>>>
>>>
>>> To Joe's point, this may not be an issue in the upcoming 1.5.0 release
>>> as it may have been fixed under [1].
>>>
>>>
>>>
>>> Regards,
>>>
>>> Matt
>>>
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-3472
>>>
>>> On Wed, Jan 10, 2018 at 11:14 AM, Georg Heiler <
>>> georg.kf.heiler@gmail.com> wrote:
>>>
>>> Regarding the stack trace I will clarify tomorrow. But it is pretty
>>> similar and cause from ticket renewal failure.
>>>
>>> Georg Heiler <georg.kf.heiler@gmail.com> schrieb am Mi. 10. Jan. 2018
>>> um 17:13:
>>>
>>> No. For sure some 3.0.x, but not entirely sure which one. Just realized
>>> that this must then be nifi 1.2 :(
>>>
>>> Schneider, Jonathan <jon@sclhs.net> schrieb am Mi. 10. Jan. 2018 um
>>> 17:11:
>>>
>>> HDF 3.0.0?
>>>
>>>
>>>
>>> *Jonathan Schneider*
>>>
>>> Hadoop/UNIX Administrator, STSC
>>>
>>> SCL Health
>>>
>>> 17501 W. 98
>>> <https://maps.google.com/?q=17501+W.+98&entry=gmail&source=g>th
St,
>>> Pillars 25-33
>>>
>>> Lenexa, KS  66219
>>>
>>> P: 913.895.2999 <(913)%20895-2999>
>>>
>>> jon@sclhs.net <jon@sclhs.net%20%0d>
>>>
>>> www.sclhealthsystem.org
>>>
>>>
>>>
>>>
>>>
>>> *From:* Georg Heiler [mailto:georg.kf.heiler@gmail.com]
>>>
>>> *Sent:* Wednesday, January 10, 2018 10:07 AM
>>>
>>>
>>> *To:* users@nifi.apache.org
>>> *Subject:* Re: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew
>>> tickets
>>>
>>>
>>>
>>> Hive is 1.2.1
>>>
>>> Joe Witt <joe.witt@gmail.com> schrieb am Mi. 10. Jan. 2018 um 17:04:
>>>
>>> Interesting.  Not what I thought it might have been.
>>>
>>> Can you share the following:
>>> - NiFi config details for the Hive processors and any controller services
>>> - Hive version.
>>>
>>> And then lets see if someone who knows Hive and our NiFi components
>>> for it far better than I can chime in :)
>>>
>>>
>>>
>>> On Wed, Jan 10, 2018 at 8:56 AM, Schneider, Jonathan <jon@sclhs.net>
>>> wrote:
>>> > For reference, the specific error I get is:
>>> >
>>> > 2018-01-10 09:55:55,988 ERROR [Timer-Driven Process Thread-10]
>>> o.apache.nifi.processors.hive.PutHiveQL
>>> PutHiveQL[id=3a4f82fd-015f-1000-0000-00005aa22fb2] Failed to update Hive
>>> for
>>> StandardFlowFileRecord[uuid=7ba71cdb-7557-4eab-bd2d-bd89add1c73f,claim=StandardContentClaim
>>> [resourceClaim=StandardResourceClaim[id=1515205062419-12378,
>>> container=default, section=90], offset=342160,
>>> length=247],offset=0,name=vp_employmentstat.orc,size=247] due to
>>> java.sql.SQLException: org.apache.thrift.transport.TTransportException:
>>> org.apache.http.client.ClientProtocolException; it is possible that
>>> retrying the operation will succeed, so routing to retry:
>>> java.sql.SQLException: org.apache.thrift.transport.TTransportException:
>>> org.apache.http.client.ClientProtocolException
>>> > java.sql.SQLException:
>>> org.apache.thrift.transport.TTransportException:
>>> org.apache.http.client.ClientProtocolException
>>> >         at
>>> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:308)
>>> >         at
>>> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
>>> >         at
>>> org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)
>>> >         at
>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
>>> >         at
>>> org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
>>> >         at
>>> org.apache.nifi.processors.hive.PutHiveQL.lambda$null$3(PutHiveQL.java:218)
>>> >         at
>>> org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
>>> >         at
>>> org.apache.nifi.processors.hive.PutHiveQL.lambda$new$4(PutHiveQL.java:199)
>>> >         at
>>> org.apache.nifi.processor.util.pattern.Put.putFlowFiles(Put.java:59)
>>> >         at
>>> org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:101)
>>> >         at
>>> org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:255)
>>> >         at
>>> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>>> >         at
>>> org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
>>> >         at
>>> org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:255)
>>> >         at
>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
>>> >         at
>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>>> >         at
>>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>> >         at
>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>>> >         at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> >         at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> >         at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> >         at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> >         at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> >         at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> >         at java.lang.Thread.run(Thread.java:745)
>>> > Caused by: org.apache.thrift.transport.TTransportException:
>>> org.apache.http.client.ClientProtocolException
>>> >         at
>>> org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
>>> >         at
>>> org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
>>> >         at
>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
>>> >         at
>>> org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
>>> >         at
>>> org.apache.hive.service.cli.thrift.TCLIService$Client.send_ExecuteStatement(TCLIService.java:223)
>>> >         at
>>> org.apache.hive.service.cli.thrift.TCLIService$Client.ExecuteStatement(TCLIService.java:215)
>>> >         at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
>>> >         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >         at java.lang.reflect.Method.invoke(Method.java:498)
>>> >         at
>>> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1374)
>>> >         at com.sun.proxy.$Proxy174.ExecuteStatement(Unknown Source)
>>> >         at
>>> org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:299)
>>> >         ... 24 common frames omitted
>>> > Caused by: org.apache.http.client.ClientProtocolException: null
>>> >         at
>>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
>>> >         at
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
>>> >         at
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
>>> >         at
>>> org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
>>> >         ... 35 common frames omitted
>>> > Caused by: org.apache.http.HttpException: null
>>> >         at
>>> org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:86)
>>> >         at
>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
>>> >         at
>>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:183)
>>> >         at
>>> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
>>> >         at
>>> org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:85)
>>> >         at
>>> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
>>> >         at
>>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
>>> >         ... 38 common frames omitted
>>> > Caused by: org.apache.http.HttpException: null
>>> >         at
>>> org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:68)
>>> >         at
>>> org.apache.hive.jdbc.HttpRequestInterceptorBase.process(HttpRequestInterceptorBase.java:74)
>>> >         ... 44 common frames omitted
>>> > Caused by: java.lang.reflect.UndeclaredThrowableException: null
>>> >         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1884)
>>> >         at
>>> org.apache.hive.service.auth.HttpAuthUtils.getKerberosServiceTicket(HttpAuthUtils.java:83)
>>> >         at
>>> org.apache.hive.jdbc.HttpKerberosRequestInterceptor.addHttpAuthHeader(HttpKerberosRequestInterceptor.java:62)
>>> >         ... 45 common frames omitted
>>> > Caused by: org.ietf.jgss.GSSException: No valid credentials provided
>>> (Mechanism level: Failed to find any Kerberos tgt)
>>> >         at
>>> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>>> >         at
>>> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
>>> >         at
>>> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>>> >         at
>>> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
>>> >         at
>>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>>> >         at
>>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>>> >         at
>>> org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:183)
>>> >         at
>>> org.apache.hive.service.auth.HttpAuthUtils$HttpKerberosClientAction.run(HttpAuthUtils.java:151)
>>> >         at java.security.AccessController.doPrivileged(Native Method)
>>> >         at javax.security.auth.Subject.doAs(Subject.java:422)
>>> >         at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
>>> >         ... 47 common frames omitted
>>> >
>>> > Jonathan Schneider
>>> > Hadoop/UNIX Administrator, STSC
>>> > SCL Health
>>> > 17501 W. 98th St
>>> <https://maps.google.com/?q=17501+W.+98th+St&entry=gmail&source=g>,
>>> Pillars 25-33
>>> > Lenexa, KS  66219
>>> > P: 913.895.2999 <(913)%20895-2999>
>>> > jon@sclhs.net
>>> > www.sclhealthsystem.org
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: Joe Witt [mailto:joe.witt@gmail.com]
>>> > Sent: Wednesday, January 10, 2018 9:55 AM
>>> > To: users@nifi.apache.org
>>> > Subject: Re: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew tickets
>>> >
>>> > Cool.  This is probably fixed in Apache NiFi 1.5.0 but please share
>>> the stack dump when it is stuck.
>>> >
>>> > bin/nifi.sh dump
>>> >
>>> > Then send us the logs dir content.
>>> >
>>> > Thanks
>>> >
>>> > On Wed, Jan 10, 2018 at 8:54 AM, Schneider, Jonathan <jon@sclhs.net>
>>> wrote:
>>> >> Joe,
>>> >>
>>> >> I can reproduce this easily.  Set up a connection to a kerberized
>>> Hive instance.  After 24 hours you will get errors about an expired TGT.
>>> Restarting the Ni-Fi process is the only way I've found to get it to renew
>>> the TGT.
>>> >>
>>> >> Jonathan Schneider
>>> >> Hadoop/UNIX Administrator, STSC
>>> >> SCL Health
>>> >> 17501 W. 98th St
>>> <https://maps.google.com/?q=17501+W.+98th+St&entry=gmail&source=g>,
>>> Pillars 25-33
>>> >> Lenexa, KS  66219
>>> >> P: 913.895.2999 <(913)%20895-2999>
>>> >> jon@sclhs.net
>>> >> www.sclhealthsystem.org
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> -----Original Message-----
>>> >> From: Joe Witt [mailto:joe.witt@gmail.com]
>>> >> Sent: Wednesday, January 10, 2018 9:53 AM
>>> >> To: users@nifi.apache.org
>>> >> Subject: [EXTERNAL EMAIL]Re: Kerberos hive failure to renew tickets
>>> >>
>>> >> *** CAUTION!  This email came from outside SCL Health. Do not open
>>> >> attachments or click links if you do not recognize the sender. ***
>>> >>
>>> >> Georg
>>> >>
>>> >> We'd need to see what you mean to really understand.  Can you please
>>> share NiFi logs directory content and if the flow is stuck/locked up please
>>> share a nifi thread dump which will be in the logs if you first run
>>> bin/nifi.sh dump.
>>> >>
>>> >> thanks
>>> >>
>>> >> On Wed, Jan 10, 2018 at 8:50 AM, Georg Heiler <
>>> georg.kf.heiler@gmail.com> wrote:
>>> >>> Hi
>>> >>> In production I observe problems with ticket renewal for the nifi
>>> >>> hive processor.
>>> >>>
>>> >>> A workaround is to restart the hive service but that doesn't seem
>>> right.
>>> >>>
>>> >>> Is there a real fix for this problem?
>>> >>>
>>> >>> Best Georg
>>> >
>>>
>>>
>

Mime
View raw message