Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 96745 invoked from network); 1 Mar 2011 14:42:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 1 Mar 2011 14:42:48 -0000 Received: (qmail 75247 invoked by uid 500); 1 Mar 2011 14:42:45 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 74807 invoked by uid 500); 1 Mar 2011 14:42:40 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 74799 invoked by uid 99); 1 Mar 2011 14:42:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 14:42:40 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of eternalyouth@gmail.com designates 209.85.214.48 as permitted sender) Received: from [209.85.214.48] (HELO mail-bw0-f48.google.com) (209.85.214.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Mar 2011 14:42:33 +0000 Received: by bwz8 with SMTP id 8so5985063bwz.35 for ; Tue, 01 Mar 2011 06:42:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:reply-to:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=Cs8g1LMHgGd6JuB9Zx1vlejnFHkvSu+aY6riXjsqYDo=; b=XOADRGSPdbLfPPMO9dz85W5tRm9djlZlZV45teQvlNQ+f2WKrB3NMoL0CbPQ3UjPa1 OTTEKDjWI6NzcMfDVcsT84KaQRGGJiIMsTIenOWBYamlke1eonNy8Nx/AkH0ulqOQikc Zr1w4J/OkgDijCe2VfEoImdlRIl93LC03k+5A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; b=Jx+pHWbbq/h9JrrXhchtyHxTOD73lMO3WIZVfsW/QAmoEyoaPib9bGKeJ9HoC4j5wO kA48NaBSn+UCatAiAssl5qcE2gc0LHTJXCwBrAmTsOrBOOnPwLE0xKhE1rPsrK9mFOfq vVE9c4EKivA8JjGMekoyKQxKJMTAouJptFqao= MIME-Version: 1.0 Received: by 10.204.140.70 with SMTP id h6mr5982380bku.117.1298990533542; Tue, 01 Mar 2011 06:42:13 -0800 (PST) Received: by 10.204.14.206 with HTTP; Tue, 1 Mar 2011 06:42:13 -0800 (PST) Reply-To: eternalyouth@gmail.com In-Reply-To: <4D6D023E.10408@tum.de> References: <4D6D023E.10408@tum.de> Date: Tue, 1 Mar 2011 15:42:13 +0100 Message-ID: Subject: Re: disable pipelining in Hadoop From: Bibek Paudel To: common-user@hadoop.apache.org Cc: Benjamin Gufler Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Tue, Mar 1, 2011 at 3:27 PM, Benjamin Gufler wr= ote: > Hi Bikash, > > On 2011-03-01 15:13, bikash sharma wrote: >> >> Is there a way to disable the use of pipelining , i.e., the reduce phase >> is >> started only after the map phase is completed? > > you need to configure the mapred.reduce.slowstart.completed.maps property= in > mapred-site.xml. It gives the percentage of mappers which must be complet= e > before the first reducers are launched. By setting it to 1, you should > obtain the wanted behaviour. > I think this only schedules the reducers, and the scheduled reducers start "copy" (followed by sort) stages. The actual "reduce" functions are called only after all the intermediate data from all mappers have been copied over. -b > Cheers, > =A0 =A0 =A0 =A0Benjamin >