Return-Path: Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: (qmail 21282 invoked from network); 14 Jul 2010 19:06:51 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 14 Jul 2010 19:06:51 -0000 Received: (qmail 2250 invoked by uid 500); 14 Jul 2010 19:06:51 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 2191 invoked by uid 500); 14 Jul 2010 19:06:50 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 2183 invoked by uid 99); 14 Jul 2010 19:06:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:06:50 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yuzhihong@gmail.com designates 209.85.160.176 as permitted sender) Received: from [209.85.160.176] (HELO mail-gy0-f176.google.com) (209.85.160.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Jul 2010 19:06:44 +0000 Received: by gyg10 with SMTP id 10so114359gyg.35 for ; Wed, 14 Jul 2010 12:06:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=KRcfQZr5xJMtrZjFV1dtW7QKud72in/9fI8kYN63oqQ=; b=Xb8h0D/LT1DS1G6bWEMZ5+z+N3kTXjB4+a8IJLgUTpU70PGW0qLOsSoJl2c0LpSCHe ghmeuzbQNc1Ltg+mHWp3dR5fiEqSX2UX/KCgxEEkHw/Lcp6LYjIrHPN5TECnoiUW4OAY Mk9W+lnIDC2MMP4gldtXg9jyTI0pr1XeFmfno= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=fXAchiUObQxcYTg3qt4DfmjVm28SfMcLgu1L8PlwU5FKhyGVYtiT4zrc8b+ko+yfLb A0WAzDSPNJIBDGsgomFx0lIK6Vn0hakt6yBlxs7TZn2WrjRdiRLWmqbouDfPWPcEiJRE XF/B5NVCbVH8w/CTVCJ7ISqS1BWzTs9nlwc+8= MIME-Version: 1.0 Received: by 10.101.139.29 with SMTP id r29mr18974869ann.92.1279134382194; Wed, 14 Jul 2010 12:06:22 -0700 (PDT) Received: by 10.101.4.24 with HTTP; Wed, 14 Jul 2010 12:06:22 -0700 (PDT) In-Reply-To: <594FA796-0841-464D-BB67-2CB1CE553962@linkedin.com> References: <594FA796-0841-464D-BB67-2CB1CE553962@linkedin.com> Date: Wed, 14 Jul 2010 12:06:22 -0700 Message-ID: Subject: Re: specify different number of mapper tasks for different machines From: Ted Yu To: mapreduce-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016e6d3cf22788618048b5daed5 X-Virus-Checked: Checked by ClamAV on apache.org --0016e6d3cf22788618048b5daed5 Content-Type: text/plain; charset=ISO-8859-1 hadoop-daemon.sh also needs to be modified - it would wipe your custom config files: if [ "$HADOOP_MASTER" != "" ]; then echo rsync from $HADOOP_MASTER rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' $HADOOP_MASTER/ "$HADOOP_HOME" fi On Wed, Jul 14, 2010 at 11:53 AM, Allen Wittenauer wrote: > > On Jul 14, 2010, at 11:50 AM, Shaojun Zhao wrote: > > Is there any way to specify that some machines run, say, 8 mapper > > tasks, while some machines run only 2 tasks? > > A custom mapred-site.xml per machine. > > > --0016e6d3cf22788618048b5daed5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable hadoop-daemon.sh also needs to be modified - it would wipe your custom conf= ig files:
=A0=A0=A0 if [ "$HADOOP_MASTER" !=3D "" ];= then
=A0=A0=A0=A0=A0 echo rsync from $HADOOP_MASTER
=A0=A0=A0=A0=A0 = rsync -a -e ssh --delete --exclude=3D.svn --exclude=3D'logs/*' --ex= clude=3D'contrib/hod/logs/*' $HADOOP_MASTER/ "$HADOOP_HOME&quo= t;
=A0=A0=A0 fi

On Wed, Jul 14, 2010 at 11:5= 3 AM, Allen Wittenauer <awittenauer@linkedin.com> wrote:

On Jul 14, 2010, at 11:50 AM, Shaojun Zhao wrote:
> Is there any way to specify that some machines run, say, 8 mapper
> tasks, while some machines run only 2 tasks?

A custom mapred-site.xml per machine.



--0016e6d3cf22788618048b5daed5--