hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "takeshi.miao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10379) Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level ERROR to WARN
Date Thu, 23 Jan 2014 03:28:37 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879416#comment-13879416

takeshi.miao commented on HBASE-10379:

I did some test on +new version of Krb5-server (1.10.3), and seems this issue is mitigated+.
I have been using 20-threads of HBase Client to put 7 records into one HTable with corresponding
7 regions scattered on three regionservers, with three round of tests; this simple test can
simply reproduce this 'Request is a replay' issue on the first environment (HBase-0.94.16
with krb5-server-1.6.1 on CentOS-5.3); But in second environment (HBase-0.94.16 with krb5-server-1.10.3
on RHEL-6.4_GA-x86_64-10-Hourly2 on AWS), I did not reproduce this issue. It seems that this
issue was fixed in the later of krb5-server release.

I did this test further on new version of krb5-server is due to I googled two tickets ([#5924|http://krbdev.mit.edu/rt/Ticket/Display.html?id=5924]
& [#1201|http://krbdev.mit.edu/rt/Ticket/Display.html?id=1201]) whom were talking about
'_KRB5KRB_AP_ERR_REPEAT_' bug in krb5 community.

> Turn the msg "Request is a replay (34) - PROCESS_TGS" from logging level ERROR to WARN
> --------------------------------------------------------------------------------------
>                 Key: HBASE-10379
>                 URL: https://issues.apache.org/jira/browse/HBASE-10379
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 0.94.16
>            Reporter: takeshi.miao
>            Assignee: takeshi.miao
>            Priority: Minor
> Hi All,
> Recently we got the error msg "Request is a replay (34) - PROCESS_TGS" while we are using
the HBase client API to put data into HBase-0.94.16 with krb5-1.6.1 enabled. The related msg
as follows...
> {code}
> [2014-01-15 09:40:38,452][hbase-tablepool-1-thread-3][ERROR][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1124)):
PriviledgedActionException as:takeshi_miao@LAB cause:javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Request
is a replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.logPriviledgedAction(UserGroupInformation.java:1143)):
PriviledgedAction as:takeshi_miao@LAB from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> [2014-01-15 09:40:38,453][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.ipc.SecureClient](org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:213)):
Exception encountered while connecting to the server : javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level:
Request is a replay (34) - PROCESS_TGS)]
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:657)):
Initiating logout for takeshi_miao@LAB
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.logout(UserGroupInformation.java:154)):
hadoop logout
> [2014-01-15 09:40:38,454][hbase-tablepool-1-thread-3][INFO ][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation.reloginFromTicketCache(UserGroupInformation.java:667)):
Initiating re-login for takeshi_miao@LAB
> [2014-01-15 09:40:38,455][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.login(UserGroupInformation.java:146)):
hadoop login
> [2014-01-15 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:95)):
hadoop login commit
> [2014-01-15 09:40:38,456][hbase-tablepool-1-thread-3][DEBUG][org.apache.hadoop.security.UserGroupInformation](org.apache.hadoop.security.UserGroupInformation$HadoopLoginModule.commit(UserGroupInformation.java:100)):
using existing subject:[takeshi_miao@LAB, UnixPrincipal: takeshi_miao, UnixNumericUserPrincipal:
501, UnixNumericGroupPrincipal [Primary Group]: 501, UnixNumericGroupPrincipal [Supplementary
Group]: 502, takeshi_miao@LAB]
> {code}
> At the beginning, we were worry about the "data loss" occurring while we found the "Request
is a replay (34) - PROCESS_TGS" (especially it is the ERROR level) in log, but after code
study, this is basically *NOT* a data loss issue due to HBase client API would try 5 times
internally (o.a.h.hbase.ipc.SecureClient, L#296, a one thread) and also 10 times of retry
externally (o.a.h.hbase.client.HConnectionManager, L#1661, for all failed thread), The HTable
API would also throw IOEcxeption to client code if any thread still fail after these retries.
> Based on HBase users' viewpoint as us, we think this is better to change the logging
level from 'ERROR' to 'WARN', due to the 'ERROR' level had been confused us for a while...But
this code change may need to change boht HBase code and Hadoop code as well; so I am wondering
how community think about this small thing but may be important to the pure HBase users.
> mailing list
> http://mail-archives.apache.org/mod_mbox/hbase-user/201401.mbox/%3CCADcMMgGiEyho0HGwgbfOUS78ymDpCo5Q0PStWAPUk40W%3DPfcFQ%40mail.gmail.com%3E

This message was sent by Atlassian JIRA

View raw message