From hadoop-dev-return-19-apmail-lucene-hadoop-dev-archive=lucene.apache.org@lucene.apache.org Mon Feb 06 18:42:22 2006 Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 48227 invoked from network); 6 Feb 2006 18:42:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 6 Feb 2006 18:42:22 -0000 Received: (qmail 21811 invoked by uid 500); 6 Feb 2006 18:42:21 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 21795 invoked by uid 500); 6 Feb 2006 18:42:21 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 21779 invoked by uid 99); 6 Feb 2006 18:42:21 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [192.87.106.226] (HELO ajax.apache.org) (192.87.106.226) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Feb 2006 10:42:19 -0800 Received: from ajax.apache.org (ajax.apache.org [127.0.0.1]) by ajax.apache.org (Postfix) with ESMTP id 30F45DF for ; Mon, 6 Feb 2006 19:41:58 +0100 (CET) Message-ID: <1492002121.1139251318198.JavaMail.jira@ajax.apache.org> Date: Mon, 6 Feb 2006 19:41:58 +0100 (CET) From: "Doug Cutting (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Moved: (HADOOP-9) mapred.local.dir temp dir. space allocation limited by smallest area MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-9?page=all ] Doug Cutting moved NUTCH-181 to HADOOP-9: ----------------------------------------- Project: Hadoop (was: Nutch) Key: HADOOP-9 (was: NUTCH-181) Version: (was: 0.8-dev) > mapred.local.dir temp dir. space allocation limited by smallest area > --------------------------------------------------------------------- > > Key: HADOOP-9 > URL: http://issues.apache.org/jira/browse/HADOOP-9 > Project: Hadoop > Type: Bug > Components: indexer > Environment: all > Reporter: Paul Baclace > > When mapred.local.dir is used to specify multiple temp dir. areas, space allocation limited by smallest area because the temp dir. selection algorithm is "round robin starting from a randomish point". When round robin is used with approximately constant sized chunks, the smallest area runs out of space first, and this is a fatal error. > Workaround: only list local fs dirs in mapred.local.dir with similarly-sized available areas. > I wrote a patch to JobConf (currenly being tested) which uses df to check available space (once a minute or less often) and then uses an efficient roulette selection to do allocation weighted by magnitude of available space. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira