hawq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lav Jain (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HAWQ-1622) Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each request
Date Thu, 26 Jul 2018 19:03:00 GMT

    [ https://issues.apache.org/jira/browse/HAWQ-1622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558775#comment-16558775
] 

Lav Jain edited comment on HAWQ-1622 at 7/26/18 7:02 PM:
---------------------------------------------------------

Evaluation notes on caching options

1. LRU/timed expiration isn't appropriate because someone might be holding a reference to
the UGI object when it's evicted from the cache. Existing java caching libraries (Guava,JCS,EHCache)
dont' allow us to implement a policy which leverages both reference and expiry.

2. A WeakReference-based cache wouldn't work because we need fine-grained control over when
UGIs are removed from the cache.


was (Author: lavjain):
We can limit the number of UGIs we create by caching them for the duration of the session

1. LRU/timed expiration isn't appropriate because someone might be holding a reference to
the UGI object when it's evicted from the cache. Existing java caching libraries (Guava,JCS,EHCache)
dont' allow us to implement a policy which leverages both reference and expiry.

2. A WeakReference-based cache wouldn't work because we need fine-grained control over when
UGIs are removed from the cache.

> Cache PXF proxy UGI so that cleanup of FileSystem cache doesn't have to be done on each
request
> -----------------------------------------------------------------------------------------------
>
>                 Key: HAWQ-1622
>                 URL: https://issues.apache.org/jira/browse/HAWQ-1622
>             Project: Apache HAWQ
>          Issue Type: Improvement
>          Components: PXF
>            Reporter: Alexander Denissov
>            Assignee: Lav Jain
>            Priority: Major
>
> Closing PXF proxy UGIs on each request (implemented in HAWQ-1621) slows down PXF request
response time significantly when several threads work concurrently as it locks FileSystem
cache and holds the lock while the cleanup of DFSClients is completed.
> This can be avoided by caching the proxy UGI for a given proxy user between requests.
Care must be taken to remove the cached entry after some pre-defined TTL if and only if there
are no current threads using any FileSystem entries held by the cache. A combination of TTL-based
cache with ref-counting might be utilized to achieve this.
>  
> For some example of this, see: 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/UserGroupInformationService.java
> Caching UGIs might be tricky when Kerberos support is implemented later, see: https://issues.apache.org/jira/browse/HIVE-3098?focusedCommentId=13398979&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13398979



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message