Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 47156 invoked from network); 5 Jan 2011 23:27:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Jan 2011 23:27:23 -0000 Received: (qmail 90555 invoked by uid 500); 5 Jan 2011 23:27:21 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 90482 invoked by uid 500); 5 Jan 2011 23:27:21 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 90474 invoked by uid 500); 5 Jan 2011 23:27:21 -0000 Delivered-To: apmail-hadoop-core-user@hadoop.apache.org Received: (qmail 90471 invoked by uid 500); 5 Jan 2011 23:27:20 -0000 Delivered-To: apmail-lucene-hadoop-user@lucene.apache.org Received: (qmail 90468 invoked by uid 99); 5 Jan 2011 23:27:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 23:27:20 +0000 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=FREEMAIL_FROM,RFC_ABUSE_POST,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 216.139.236.26 is neither permitted nor denied by domain of marc.sturlese@gmail.com) Received: from [216.139.236.26] (HELO sam.nabble.com) (216.139.236.26) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jan 2011 23:27:14 +0000 Received: from ben.nabble.com ([192.168.236.152]) by sam.nabble.com with esmtp (Exim 4.69) (envelope-from ) id 1Packn-00042y-HB for hadoop-user@lucene.apache.org; Wed, 05 Jan 2011 15:26:53 -0800 Date: Wed, 5 Jan 2011 15:26:53 -0800 (PST) From: Marc Sturlese To: hadoop-user@lucene.apache.org Message-ID: <1294270013529-2202221.post@n3.nabble.com> In-Reply-To: References: <1294245341956-2199517.post@n3.nabble.com> Subject: Re: LocalDirAllocator and getLocalPathForWrite MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hey Todd, >>LocalDirAllocator is an internal-facing API and you shouldn't be using it >>from user code. If you write into mapred.local.dir like this, you will end >>up with conflicts between different tasks running from the same node I know it's a bit odd usage but the thing is that I need to create files in the local file system, work in there with them amb after that upload them to hdfs (I use the outputcomitter.) To avoid the conflicts you talk about, I create a folder which looks like "mapred.local.dir"/taskId/attemptId and I work there and aparently I am having no problems. >>and there isn't usually a good reason to write to multiple drives from >>within a task When I said I had a cluster of one node, was just to try to clarify my doubt and explain the example. My cluster is bigger than that actually and each node has more than 1 phisical disk. To have multuple task running at the same time is what I do. I would like each task to write just to a single local disk but don't know how to do it. >>The working directory of your MR task is already within one of the drives, Is there a way to get a working directory in the local disk from the reducer? Could I do something similar to: FileSystem fs = FileSystem.get(conf); LocalFileSystem localFs = fs.getLocal(conf); Path path = localFs.getWorkingDirectory(); I would apreciate if you can tell me a bit more about this. I need to deal with these files just in local and want them copied to hdfs just when I finish working with them. Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/LocalDirAllocator-and-getLocalPathForWrite-tp2199517p2202221.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com.