hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5369) Support negative caching of user-group mapping
Date Wed, 25 Jun 2014 23:44:25 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Lei (Eddy) Xu updated HDFS-5369:

    Attachment: HDFS-5369.000.patch

This patch re-enables the negative cache behavior in user-group mapping, so that it would
not result _*long-time*_ frequently retries when the user/group resolution service has transient
issues. The difference between this patch and HADOOP-8088 is that this patch adds another
timeout for negative cached items. Thus it should be able to differentiate the expiring times
for normal case and negative cached case. 

It also reduces the error message generated from Ldap and Shell based GroupsMappings. 

It would be great to see whether this patch can fit into different cases from [~andrew.wang],
[~vinayrpet] and [~kihwal]. All feedbacks are wellcome :)

> Support negative caching of user-group mapping
> ----------------------------------------------
>                 Key: HDFS-5369
>                 URL: https://issues.apache.org/jira/browse/HDFS-5369
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.2.0
>            Reporter: Andrew Wang
>         Attachments: HDFS-5369.000.patch
> We've seen a situation at a couple of our customers where interactions from an unknown
user leads to a high-rate of group mapping calls. In one case, this was happening at a rate
of 450 calls per second with the shell-based group mapping, enough to severely impact overall
namenode performance and also leading to large amounts of log spam (prints a stack trace each
> Let's consider negative caching of group mapping, as well as quashing the rate of this
log message.

This message was sent by Atlassian JIRA

View raw message