Return-Path: Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: (qmail 2761 invoked from network); 2 Oct 2010 05:15:56 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 2 Oct 2010 05:15:56 -0000 Received: (qmail 9701 invoked by uid 500); 2 Oct 2010 05:15:56 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 9596 invoked by uid 500); 2 Oct 2010 05:15:54 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 9586 invoked by uid 99); 2 Oct 2010 05:15:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 05:15:53 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.22] (HELO thor.apache.org) (140.211.11.22) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 05:15:53 +0000 Received: from thor (localhost [127.0.0.1]) by thor.apache.org (8.13.8+Sun/8.13.8) with ESMTP id o925FX0X005530 for ; Sat, 2 Oct 2010 05:15:33 GMT Message-ID: <27944089.510121285996533261.JavaMail.jira@thor> Date: Sat, 2 Oct 2010 01:15:33 -0400 (EDT) From: "Ramkumar Vadali (JIRA)" To: mapreduce-issues@hadoop.apache.org Subject: [jira] Updated: (MAPREDUCE-1819) RaidNode should be smarter in submitting Raid jobs In-Reply-To: <20336598.8631274908299303.JavaMail.jira@thor> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali updated MAPREDUCE-1819: --------------------------------------- Status: Patch Available (was: Open) > RaidNode should be smarter in submitting Raid jobs > -------------------------------------------------- > > Key: MAPREDUCE-1819 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1819 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/raid > Affects Versions: 0.20.1 > Reporter: Ramkumar Vadali > Assignee: Ramkumar Vadali > Attachments: MAPREDUCE-1819.4.patch, MAPREDUCE-1819.patch, MAPREDUCE-1819.patch.2, MAPREDUCE-1819.patch.3 > > > The RaidNode currently computes parity files as follows: > 1. Using RaidNode.selectFiles() to figure out what files to raid for a policy > 2. Using #1 repeatedly for each configured policy to accumulate a list of files. > 3. Submitting a mapreduce job with the list of files from #2 using DistRaid.doDistRaid() > This task addresses the fact that #2 and #3 happen sequentially. The proposal is to submit a separate mapreduce job for the list of files for each policy and use another thread to track the progress of the submitted jobs. This will help reduce the time taken for files to be raided. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.