Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Wed, 8 Jun 2011 16:00:59 +0000 (UTC)
From: "Nathan Roberts (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <1549498761.3555.1307548859072.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1253777271.389.1307455378937.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (MAPREDUCE-2572) Throttle the deletion of data
 from the distributed cache
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046036#comment-13046036 ] 

Nathan Roberts commented on MAPREDUCE-2572:
-------------------------------------------

One option might be to just do something very simple after we wakeup from every CheckPeriod. Rather than throttling logic that could be a little sophisticated and error prone. Maybe just something like:
- If below low water mark - do nothing
- If between low and high water marks - delete up-to a configurable number of archives (e.g. 1 is probably a nice default), or until LWM is reached
- If above high water mark, delete archives until back at low water mark 

> Throttle the deletion of data from the distributed cache
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2572
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2572
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: distributed-cache
>    Affects Versions: 0.20.205.0
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>         Attachments: THROTTLING-security-v1.patch
>
>
> When deleting entries from the distributed cache we do so in a background thread.  Once the size limit of the distributed cache is reached all unused entries are deleted.  MAPREDUCE-2494 changes this so that entries are deleted in LRU order until the usage falls below a given threshold.  In either of these cases we are periodically flooding a disk with delete requests which can slow down all IO operations to a drive.  It would be better to be able to throttle this deletion so that it is spread out over a longer period of time.  This jira is to add in this throttling.
> On investigating it seems much simpler to backport MPAREDUCE-2494 to 20S before implementing this change rather then try to implement it without LRU deletion, because LRU goes a long way towards reducing the load on the disk anyways.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira