Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <28170148.53861287987563933.JavaMail.jira@thor>
Date: Mon, 25 Oct 2010 02:19:23 -0400 (EDT)
From: "Ramkumar Vadali (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Assigned: (MAPREDUCE-1892) RaidNode can allow layered
 policies more efficiently
In-Reply-To: <20735527.26651277331411619.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/MAPREDUCE-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramkumar Vadali reassigned MAPREDUCE-1892:
------------------------------------------

    Assignee: Ramkumar Vadali

> RaidNode can allow layered policies more efficiently
> ----------------------------------------------------
>
>                 Key: MAPREDUCE-1892
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1892
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/raid
>            Reporter: Ramkumar Vadali
>            Assignee: Ramkumar Vadali
>
> The RaidNode policy file can have layered policies that can cover a file more than once. To avoid processing a file multiple times (for RAIDing), RaidNode maintains a list of processed files that is used to avoid duplicate processing attempts.
> This is problematic in that a large number of processed files could cause the RaidNode to run out of memory.
> This task proposes a better method of detecting processed files. The method is based on the observation that a more selective policy will have a better match with a file name than a less selective one. Specifically, the more selective policy will have a longer common prefix with the file name.
> So to detect if a file has already been processed, the RaidNode only needs to maintain a list of processed policies and compare the lengths of the common prefixes. If the file has a longer common prefix with one of the processed policies than with the current policy, it can be assumed to be processed already.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.