Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 96325 invoked from network); 18 Aug 2007 14:00:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 18 Aug 2007 14:00:07 -0000 Received: (qmail 60354 invoked by uid 500); 18 Aug 2007 14:00:03 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 60332 invoked by uid 500); 18 Aug 2007 14:00:03 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 60322 invoked by uid 99); 18 Aug 2007 14:00:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Aug 2007 07:00:03 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Aug 2007 13:59:56 +0000 Received: from arunc.eglbp.corp.yahoo.com (arunc.eglbp.corp.yahoo.com [10.66.74.226]) by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id l7IDxIft005594 for ; Sat, 18 Aug 2007 06:59:19 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-authentication-warning:date:from:to:subject: message-id:references:mime-version:content-type: content-disposition:in-reply-to:user-agent; b=ccrAmkisV2+ncMNBlYMsspvv23i1wVUIDWkngIUlAYd9ebYroVALjs5rMT0UPdox Received: (from arunc@localhost) by arunc.eglbp.corp.yahoo.com (8.13.3/8.13.3) id l7IDxIS2067112 for hadoop-user@lucene.apache.org; Sat, 18 Aug 2007 19:29:18 +0530 (IST) (envelope-from arunc@yahoo-inc.com) X-Authentication-Warning: arunc.eglbp.corp.yahoo.com: arunc set sender to arunc@yahoo-inc.com using -f Date: Sat, 18 Aug 2007 19:29:18 +0530 From: Arun C Murthy To: hadoop-user@lucene.apache.org Subject: Re: Reduce Performance Message-ID: <20070818135918.GA66473@yahoo-inc.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i X-Virus-Checked: Checked by ClamAV on apache.org Thorsten, On Sat, Aug 18, 2007 at 01:18:06PM +0200, Thorsten Schuett wrote: >Please correct me if I am wrong, but it looks like hadoop is using the rpc >mechanism to copy the map output files to the reduce task (in this case via >the loopback device). i Hadoop uses http to copy map output files, not rpc. >If my assumptions are correct, would it be possible to >read/access the files directly in the "one-node mode"? Please take a look at LocalJobRunner in src/org/apache/hadoop/mapred ... set the jobtracker in your config to 'local' and this happens automatically. (http://wiki.apache.org/lucene-hadoop/HowToDebugMapReducePrograms) Arun >Thanks, > Thorsten