hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From takeshi <takeshi.m...@gmail.com>
Subject Suggest that turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level from ERROR to WARN
Date Fri, 17 Jan 2014 05:57:36 GMT
Hi All,

Recently we got the error msg "Request is a replay (34) - PROCESS_TGS"
while we are using the HBase client API to put data into HBase-0.94.16 with
krb5-1.6.1 enabled. The related msg as follows...
{code}
[2014-01-15
09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
PriviledgedActionException
as:takeshi_miao@LABcause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Request is a
replay (34) - PROCESS_TGS)]
[2014-01-15
09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
PriviledgedAction
as:takeshi_miao@LABfrom:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)

[2014-01-15
09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Request is a
replay (34) - PROCESS_TGS)]
[2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
Initiating logout for takeshi_miao@LAB
[2014-01-15
09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
hadoop logout
[2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO
][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
Initiating re-login for takeshi_miao@LAB
[2014-01-15
09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
hadoop login
[2014-01-15
09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
hadoop login commit
[2014-01-15
09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao,
UnixNumericUserPrincipal: 501, UnixNumericGroupPrincipal [Primary Group]:
501, UnixNumericGroupPrincipal [Supplementary Group]: 502, takeshi_miao@LAB]
{code}

At the beginning, we were worry about the "data loss" occurring while we
found the "Request is a replay (34) - PROCESS_TGS" (especially it is the
ERROR level) in log, but after code study, this is basically *NOT* a data
loss issue due to HBase client API would try 5 times internally
(o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of
retry externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all
failed thread), The HTable API would also throw IOEcxeption to client code
if any thread still fail after these retries.

Based on HBase users' viewpoint as us, we think this is better to change
the logging level from 'ERROR' to 'WARN', due to the 'ERROR' level had been
confused us for a while...But this code change may need to change boht
HBase code and Hadoop code as well; so I am wondering how community think
about this small thing but may be important to the pure HBase users.

P.S. more details are in this article
http://dumbointaiwan.blogspot.tw/2014/01/hbase-and-keberos-request-is-replay-34.html

Tks

Best regards

takeshi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message