hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8892) ShortCircuitCache.CacheCleaner can add Slot.isInvalid() check too
Date Sat, 22 Aug 2015 15:54:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14708067#comment-14708067
] 

Colin Patrick McCabe commented on HDFS-8892:
--------------------------------------------

I think this patch might be acceptable if you introduced a separate thread to iterate over
all slots, and had that thread run only every 10 minutes (or something like that).

In general, though, I'm curious why you are interested in this?  If a replica is used, we'll
check if it's valid at that point.  If it's not used, it should eventually get timed out.

> ShortCircuitCache.CacheCleaner can add Slot.isInvalid() check too
> -----------------------------------------------------------------
>
>                 Key: HDFS-8892
>                 URL: https://issues.apache.org/jira/browse/HDFS-8892
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.7.1
>            Reporter: Ravikumar
>            Assignee: kanaka kumar avvaru
>            Priority: Minor
>
> Currently CacheCleaner thread checks only for cache-expiry times. It would be nice if
it handles an invalid-slot too in an extra-pass of evictable map…
> for(ShortCircuitReplica replica:evictable.values()) {
>      if(!scr.getSlot().isValid()) {
>         purge(replica);
>      }
> }
> //Existing code...
> int numDemoted = demoteOldEvictableMmaped(curMs);
> int numPurged = 0;
> Long evictionTimeNs = Long.valueOf(0);
> ….
> …..
> Apps like HBase can tweak the expiry/staleness/cache-size params in DFS-Client, so that
ShortCircuitReplica will never be closed except when Slot is declared invalid. 
> I assume slot-invalidation will happen during block-invalidation/deletes {Primarily triggered
by compaction/shard-takeover etc..}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message