hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From takeshi <takeshi.m...@gmail.com>
Subject Re: Suggest that turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level from ERROR to WARN
Date Mon, 20 Jan 2014 02:40:25 GMT
Hi Stack,

https://issues.apache.org/jira/browse/HBASE-10379 ticket created,
tks for your response~

Best regards

takeshi


2014/1/18 Stack <stack@duboce.net>

> Lets change it in both places.  Please file issues.  Lets try minimize the
> freakout incidents running your hbase/hadoop cluster.
> Thanks Takeshi,
> St.Ack
>
>
> On Thu, Jan 16, 2014 at 9:57 PM, takeshi <takeshi.miao@gmail.com> wrote:
>
> > Hi All,
> >
> > Recently we got the error msg "Request is a replay (34) - PROCESS_TGS"
> > while we are using the HBase client API to put data into HBase-0.94.16
> with
> > krb5-1.6.1 enabled. The related msg as follows...
> > {code}
> > [2014-01-15
> >
> >
> 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
> > PriviledgedActionException
> > as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS
> > initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Request is
> a
> > replay (34) - PROCESS_TGS)]
> > [2014-01-15
> >
> >
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
> > PriviledgedAction
> > as:takeshi_miao@LABfrom
> > :sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > [2014-01-15
> >
> >
> 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
> > Exception encountered while connecting to the server :
> > javax.security.sasl.SaslException: GSS initiate failed [Caused by
> > GSSException: No valid credentials provided (Mechanism level: Request is
> a
> > replay (34) - PROCESS_TGS)]
> > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> >
> >
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
> > Initiating logout for takeshi_miao@LAB
> > [2014-01-15
> >
> >
> 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
> > hadoop logout
> > [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
> >
> >
> ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
> > Initiating re-login for takeshi_miao@LAB
> > [2014-01-15
> >
> >
> 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
> > hadoop login
> > [2014-01-15
> >
> >
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
> > hadoop login commit
> > [2014-01-15
> >
> >
> 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
> > using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
> > UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
> > 501, UnixNumericGroupPrincipal [Supplementary Group]: 502,
> takeshi_miao@LAB
> > ]
> > {code}
> >
> > At the beginning, we were worry about the "data loss" occurring while we
> > found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
> > ERROR level) in log, but after code study, this is basically *NOT* a data
> > loss issue due to HBase client API would try 5 times internally
> > (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
> > retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
> > failed thread), The HTable API would also throw IOEcxeption to client
> code
> > if any thread still fail after these retries.
> >
> > Based on HBase users' viewpoint as us, we think this is better to change
> > the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had
> been
> > confused us for a while...But this code change may need to change boht
> > HBase code and Hadoop code as well; so I am wondering how community think
> > about this small thing but may be important to the pure HBase users.
> >
> > P.S. more details are in this article
> >
> >
> http://dumbointaiwan.blogspot.tw/2014/01/hbase-and-keberos-request-is-replay-34.html
> >
> > Tks
> >
> > Best regards
> >
> > takeshi
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message