hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Denissov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HAWQ-1622) Cache PXF proxy UGI so that cleanup of FileSystem cache does't have to be done on each request
Date Fri, 01 Jun 2018 00:11:00 GMT
Alexander Denissov created HAWQ-1622:
----------------------------------------

             Summary: Cache PXF proxy UGI so that cleanup of FileSystem cache does't have
to be done on each request
                 Key: HAWQ-1622
                 URL: https://issues.apache.org/jira/browse/HAWQ-1622
             Project: Apache HAWQ
          Issue Type: Improvement
            Reporter: Alexander Denissov
            Assignee: Radar Lei


Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down PXF request response
time significantly when several threads work concurrently as it locks FileSystem cache and
holds the lock while the cleanup of DFSClients is completed.

This can be avoided by caching the proxy UGI for a given proxy user between requests. Care
must be taken to remove the cached entry after some pre-defined TTL if and only if there are
no current threads using any FileSystem entries held by the cache. A combination of TTL-based
cache with ref-counting might be utilized to achieve this.

 

For some example of this, see: 

https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java

Caching UGIs might be tricky when Kerberos support is implemented later, see: https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message