hawq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lav Jain (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAWQ-1622) Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each request
Date Thu, 26 Jul 2018 19:02:00 GMT

    [ https://issues.apache.org/jira/browse/HAWQ-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558778#comment-16558778

Lav Jain commented on HAWQ-1622:

Maintain reference count for each cache entry (UGI)

UGI stays in the cache when it has an active reference

UGIs expire 15 minutes after last access

Expired UGIs are cleaned up when the cache is accessed.

Clean up UGI resources immediately after processing the last block of data for a segment within

> Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each
> -----------------------------------------------------------------------------------------------
>                 Key: HAWQ-1622
>                 URL: https://issues.apache.org/jira/browse/HAWQ-1622
>             Project: Apache HAWQ
>          Issue Type: Improvement
>          Components: PXF
>            Reporter: Alexander Denissov
>            Assignee: Lav Jain
>            Priority: Major
> Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down PXF request
response time significantly when several threads work concurrently as it locks FileSystem
cache and holds the lock while the cleanup of DFSClients is completed.
> This can be avoided by caching the proxy UGI for a given proxy user between requests.
Care must be taken to remove the cached entry after some pre-defined TTL if and only if there
are no current threads using any FileSystem entries held by the cache. A combination of TTL-based
cache with ref-counting might be utilized to achieve this.
> For some example of this, see: 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java
> Caching UGIs might be tricky when Kerberos support is implemented later, see: https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979

This message was sent by Atlassian JIRA

View raw message