Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B2243E2B2 for ; Tue, 22 Jan 2013 02:27:46 +0000 (UTC) Received: (qmail 71834 invoked by uid 500); 22 Jan 2013 02:27:42 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 71750 invoked by uid 500); 22 Jan 2013 02:27:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 71740 invoked by uid 99); 22 Jan 2013 02:27:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jan 2013 02:27:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of satbeer.lamba@gmail.com designates 74.125.82.178 as permitted sender) Received: from [74.125.82.178] (HELO mail-we0-f178.google.com) (74.125.82.178) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Jan 2013 02:27:35 +0000 Received: by mail-we0-f178.google.com with SMTP id x48so257862wey.23 for ; Mon, 21 Jan 2013 18:27:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=nJFUJVQjavWbqcUetLTre3Ps66zpABUk9FqeFuJ5mVw=; b=ZH/Tpq9bPhU0OJHYb47c4/kh83aGTEnLiRuU+YUnqHd8UoGVfhibz/QDXzFkpJlj0B lbrMdHABDO6ulC8mpnmtfFxZ6S4hlgfdERyMdoy2GHLHCghMZD+iEBtVFNB1bSS7PTzt Lq+CLLM6/k5J9GXToKAwgPmNKnobayYAHSx++2Urp81+Ecrfxf2EYC/G1QCfZ45eeboi KhTD4AccJ0Du6qXUP6pg9GBVVRXPJLTw2lF2WtHiASpMPoG7tnBejv2QX2kbEROkhxFC EZILKqLvXxhf49OIcAtECNvRmllQ76hnDDvHTe3tItPYZXpiEPyh33sjsGTY/rJw+M/5 +lBg== MIME-Version: 1.0 X-Received: by 10.180.95.200 with SMTP id dm8mr18477210wib.16.1358821633980; Mon, 21 Jan 2013 18:27:13 -0800 (PST) Received: by 10.194.36.100 with HTTP; Mon, 21 Jan 2013 18:27:13 -0800 (PST) Received: by 10.194.36.100 with HTTP; Mon, 21 Jan 2013 18:27:13 -0800 (PST) In-Reply-To: References: Date: Tue, 22 Jan 2013 07:57:13 +0530 Message-ID: Subject: Re: passing arguments to hadoop job From: Satbeer Lamba To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d04430606ce78cf04d3d750ff X-Virus-Checked: Checked by ClamAV on apache.org --f46d04430606ce78cf04d3d750ff Content-Type: text/plain; charset=ISO-8859-1 Just add System.out.println (baseCount); To check if expected value is getting displayed. If not then the value is not set propely in jobconf. Also you don't need to remove static. Sorry about that, it doesn't change the expected output, though it's not required. On Jan 22, 2013 7:34 AM, "jamal sasha" wrote: > Hi, > Thanks for taking out time. > Will this not work : conf.setInt("basecount",200000); ?? > I am not sure how to add loger or syso (new to both java and hadoop :( ) > > > > On Mon, Jan 21, 2013 at 5:55 PM, Satbeer Lamba wrote: > >> Please be sure that you are getting the value of baseSum in reducer by >> adding a logger or syso. >> >> Also consider removing static in declaration of baseSum as it would add >> counts of previous keys. >> On Jan 22, 2013 7:17 AM, "jamal sasha" wrote: >> >>> The second one. >>> If the word hello appears once, its count is 2000001. >>> :) >>> >>> >>> On Mon, Jan 21, 2013 at 5:40 PM, Satbeer Lamba wrote: >>> >>>> Do you mean to say you want to count the words from 200000 th line >>>> onwards? >>>> >>>> OR >>>> >>>> You want to start counting from 2000000? >>>> For example if HELLO appears once it's count is 2000001. >>>> >>>> Please clarify >>>> On Jan 22, 2013 5:22 AM, "jamal sasha" wrote: >>>> >>>>> Hi, >>>>> Lets say I have the standard helloworld program >>>>> >>>>> http://hadoop.apache.org/docs/r0.17.0/mapred_tutorial.html#Example%3A+WordCount+v2.0 >>>>> >>>>> Now, lets say, I want to start the counting not from zero but from >>>>> 200000. >>>>> So my reference line is 200000. >>>>> >>>>> I modified the Reduce code as following: >>>>> public static class Reduce extends MapReduceBase implements >>>>> Reducer { >>>>> *private static int baseSum ;* >>>>> * public void configure(JobConf job){* >>>>> * baseSum = Integer.parseInt(job.get("basecount"));* >>>>> * * >>>>> * }* >>>>> public void reduce(Text key, Iterator values, >>>>> OutputCollector output, Reporter reporter) throws >>>>> IOException { >>>>> int sum =* baseSum*; >>>>> while (values.hasNext()) { >>>>> sum += values.next().get(); >>>>> } >>>>> output.collect(key, new IntWritable(sum)); >>>>> } >>>>> } >>>>> >>>>> >>>>> And in main added: >>>>> conf.setInt("basecount",200000); >>>>> >>>>> >>>>> >>>>> So my hope was this should have done the trick.. >>>>> But its not working. the code is running normally :( >>>>> How do i resolve this? >>>>> Thanks >>>>> >>>> >>> > --f46d04430606ce78cf04d3d750ff Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Just add
System.out.println (baseCount);
To check if expected value is getting displayed. If not then the value is n= ot set propely in jobconf.

Also you don't need to remove static. Sorry about that, it doesn'= ;t change the expected output, though it's not required.

On Jan 22, 2013 7:34 AM, "jamal sasha"= <jamalshasha@gmail.com>= wrote:
Hi,
=A0 Thanks for taking out time.=A0
Will = this not work :=A0conf.setInt("basecount",200000); ??
I am not sure how to add loger or sys= o (new to both java and hadoop :( )



On Mon, Jan 21, 2013 at 5:55 PM, Satbeer Lam= ba <satbeer.lamba@gmail.com> wrote:

Please be sure that you are getting the value of baseSum in reducer by a= dding a logger or syso.

Also consider removing static in declaration of baseSum as it would add = counts of previous keys.

On Jan 22, 2013 7:17 AM, "jamal sasha"= <jamalshasha= @gmail.com> wrote:
The second one.
If the word hello appears once, its co= unt is=A0=A02000001.
:)<= /div>


On= Mon, Jan 21, 2013 at 5:40 PM, Satbeer Lamba <satbeer.lamba@gmail.c= om> wrote:

Do you mean to say you want to count the = words from 200000 th line onwards?

OR

You want to start counting from 2000000?
For example if HELLO appears once it's count is 2000001.

Please clarify

On Jan 22, 2013 5:22 AM, "jamal sasha"= <jamalshasha= @gmail.com> wrote:
Hi,
=A0 Lets say I have the standard helloworld progra= m

Now, lets say, I want to start the counting not f= rom zero but from 200000.
So my reference line is 200000.

I modified the Reduce code as following:
=A0public static class Reduce extends MapReduceBase implements Reducer<T= ext, IntWritable, Text, IntWritable> {
=A0=A0 =A0 private static int baseSum ;
=A0=A0 =A0 public void configure(JobConf job){
=A0=A0 =A0 baseSum =3D Integer.parseInt(job.get(= "basecount"));
=A0=A0 =A0
=A0=A0 =A0 }
=A0=A0 =A0 =A0public void= reduce(Text key, Iterator<IntWritable> values, OutputCollector<Te= xt, IntWritable> output, Reporter reporter) throws IOException {
=A0=A0 =A0 =A0 =A0int sum =3D<= b>=A0baseSum;
= =A0=A0 =A0 =A0 =A0while (values.hasNext()) {
=A0=A0 =A0 =A0 =A0 =A0sum +=3D values.next().get()= ;
=A0=A0 =A0 =A0 =A0}
=
=A0=A0 =A0 =A0 =A0output.= collect(key, new IntWritable(sum));
=A0=A0 =A0 =A0}
=A0=A0 =A0}

And in main added:
=A0 =A0conf.setInt(= "basecount",200000);



=
So my hope was this should have done the trick..
But its not= working. the code is running normally :(
How do i resolve this?<= /div>
Thanks


--f46d04430606ce78cf04d3d750ff--