hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen O'Donnell (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-13263) Reload cached groups in background after expiry
Date Mon, 13 Jun 2016 21:12:03 GMT
Stephen O'Donnell created HADOOP-13263:

             Summary: Reload cached groups in background after expiry
                 Key: HADOOP-13263
                 URL: https://issues.apache.org/jira/browse/HADOOP-13263
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Stephen O'Donnell

In HADOOP-11238 the Guava cache was introduced to allow refreshes on the Namenode group cache
to run in the background, avoiding many slow group lookups. Even with this change, I have
seen quite a few clusters with issues due to slow group lookups. The problem is most prevalent
in HA clusters, where a slow group lookup on the hdfs user can fail to return for over 45
seconds causing the Failover Controller to kill it.

The way the current Guava cache implementation works is approximately:

1) On initial load, the first thread to request groups for a given user blocks until it returns.
Any subsequent threads requesting that user block until that first thread populates the cache.

2) When the key expires, the first thread to hit the cache after expiry blocks. While it is
blocked, other threads will return the old value.

I feel it is this blocking thread that still gives the Namenode issues on slow group lookups.
If the call from the FC is the one that blocks and lookups are slow, if can cause the NN to
be killed.

Guava has the ability to refresh expired keys completely in the background, where the first
thread that hits an expired key schedules a background cache reload, but still returns the
old value. Then the cache is eventually updated. This patch introduces this background reload
feature. There are two new parameters:

1) hadoop.security.groups.cache.background.reload - default false to keep the current behaviour.
Set to true to enable a small thread pool and background refresh for expired keys

2) hadoop.security.groups.cache.background.reload.threads - only relevant if the above is
set to true. Controls how many threads are in the background refresh pool. Default is 1, which
is likely to be enough.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

View raw message