hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10755) Support negative caching of user-group mapping
Date Fri, 11 Jul 2014 21:19:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14059369#comment-14059369
] 

Andrew Wang commented on HADOOP-10755:
--------------------------------------

Hi Eddy, thanks for working on this. Overall looks good, had a few review comments:

Text in core-default.xml:
* typo: nagative
* Should mention that setting a negative value disables it
* Suggested text something like this:

{quote}
Expiration time for entries in the the negative user-to-group mapping caching, in seconds.
This is useful when invalid users are retrying frequently. It is suggested to set a small
value for this expiration, since a transient error in group lookup could temporarily lock
out a legitimate user.

Set this to a negative value to disable negative user-to-group caching.
{quote}

* We have a behavior change in getGroups. Previously it would always throw an exception if
getGroups().isEmpty(), now it depends on this new config. I think the config should only control
the caching, not the return value. Would be good to add a test to enforce this behavior.
* The test looks racy, depends on the sleeps/timings to work. Any possibility of improvement
here with mocks or reaching in with @VisibleForTesting?

> Support negative caching of user-group mapping
> ----------------------------------------------
>
>                 Key: HADOOP-10755
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10755
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 2.2.0
>            Reporter: Andrew Wang
>            Assignee: Lei (Eddy) Xu
>         Attachments: HADOOP-10755.000.patch, HADOOP-10755.001.patch, HADOOP-10755.002.patch,
HDFS-5369.000.patch
>
>
> We've seen a situation at a couple of our customers where interactions from an unknown
user leads to a high-rate of group mapping calls. In one case, this was happening at a rate
of 450 calls per second with the shell-based group mapping, enough to severely impact overall
namenode performance and also leading to large amounts of log spam (prints a stack trace each
time).
> Let's consider negative caching of group mapping, as well as quashing the rate of this
log message.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message