hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7142) Implement a 2Q eviction strategy for HDFS-6581
Date Mon, 13 Oct 2014 22:36:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170134#comment-14170134

Colin Patrick McCabe commented on HDFS-7142:

bq. The call to ceiling need not return the exact match. If you get a non-null result you
need a check for blockId and bpid match. Perhaps I misunderstood the intention.

You're right... I need to check to make sure that the bpid and block id are the same after
getting back a result from {{ceiling}}.  Fixed.

bq. dequeueNextReplicaToPersist appears to have a starvation issue. If replicas in a higher
blockPoolId keep getting added constantly a replica in a lower bpid may wait indefinitely
to get persisted. It would be good to persist replicas in the same order in which they were
originally added, you can do that with an additional set.

My mistake here was looking at the lowest value in {{replicasSortedByBlockPoolAndId}}.  It
should be looking at the lowest value in {{replicasSortedByTierAndLastUsed}}.  There's no
starvation issue if it looks at the set which is sorted by lastUsed, because oldest replicas
will get picked first.  Fixed.

bq. Same issue with numReplicasNotPersisted, it should not count all replicas in RAM. Let
me clarify the documentation on RamDiskReplicaTracker.dequeueNextReplicaToPersist.

It seems to me that the number of replicas not persisted *is* all the replicas in RAM.  So
perhaps the function needs to be renamed.  Can you clarify what this should count?

bq. Colin Patrick McCabe, any comments and updates to the patch?

Let me repost a patch fixing the first two issues pointing out.  I will wait for clarification
on any other API issues.  I'm going to mark this as targetting 2.7.

> Implement a 2Q eviction strategy for HDFS-6581
> ----------------------------------------------
>                 Key: HDFS-7142
>                 URL: https://issues.apache.org/jira/browse/HDFS-7142
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: 0002-Add-RamDiskReplica2QTracker.patch
> We should implement a 2Q or approximate 2Q eviction strategy for HDFS-6581.  It is well
known that LRU is a poor fit for scanning workloads, which HDFS may often encounter. 

This message was sent by Atlassian JIRA

View raw message