Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 407EF97CB for ; Thu, 2 Feb 2012 04:14:26 +0000 (UTC) Received: (qmail 55386 invoked by uid 500); 2 Feb 2012 04:14:17 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 54100 invoked by uid 500); 2 Feb 2012 04:13:53 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 54075 invoked by uid 99); 2 Feb 2012 04:13:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Feb 2012 04:13:44 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.214.176] (HELO mail-tul01m020-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Feb 2012 04:13:37 +0000 Received: by obbwd18 with SMTP id wd18so3411205obb.35 for ; Wed, 01 Feb 2012 20:13:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.182.117.8 with SMTP id ka8mr1082318obb.73.1328155996212; Wed, 01 Feb 2012 20:13:16 -0800 (PST) Received: by 10.182.92.230 with HTTP; Wed, 1 Feb 2012 20:13:15 -0800 (PST) X-Originating-IP: [98.198.171.77] In-Reply-To: <9C3ECE32-65E4-4250-99D6-427475581A74@gmail.com> References: <9C3ECE32-65E4-4250-99D6-427475581A74@gmail.com> Date: Wed, 1 Feb 2012 22:13:15 -0600 Message-ID: Subject: Re: Can't achieve load distribution From: Mark Kerzner To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d044786035c4d9c04b7f36a9c --f46d044786035c4d9c04b7f36a9c Content-Type: text/plain; charset=ISO-8859-1 Thanks! Mark On Wed, Feb 1, 2012 at 7:44 PM, Anil Gupta wrote: > Yes, if ur block size is 64mb. Btw, block size is configurable in Hadoop. > > Best Regards, > Anil > > On Feb 1, 2012, at 5:06 PM, Mark Kerzner wrote: > > > Anil, > > > > do you mean one block of HDFS, like 64MB? > > > > Mark > > > > On Wed, Feb 1, 2012 at 7:03 PM, Anil Gupta > wrote: > > > >> Do u have enough data to start more than one mapper? > >> If entire data is less than a block size then only 1 mapper will run. > >> > >> Best Regards, > >> Anil > >> > >> On Feb 1, 2012, at 4:21 PM, Mark Kerzner > wrote: > >> > >>> Hi, > >>> > >>> I have a simple MR job, and I want each Mapper to get one line from my > >>> input file (which contains further instructions for lengthy > processing). > >>> Each line is 100 characters long, and I tell Hadoop to read only 100 > >> bytes, > >>> > >>> > >> > job.getConfiguration().setInt("mapreduce.input.linerecordreader.line.maxlength", > >>> 100); > >>> > >>> I see that this part works - it reads only one line at a time, and if I > >>> change this parameter, it listens. > >>> > >>> However, on a cluster only one node receives all the map tasks. Only > one > >>> map tasks is started. The others never get anything, they just wait. > I've > >>> added 100 seconds wait to the mapper - no change! > >>> > >>> Any advice? > >>> > >>> Thank you. Sincerely, > >>> Mark > >> > --f46d044786035c4d9c04b7f36a9c--