hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satbeer Lamba <satbeer.la...@gmail.com>
Subject Re: passing arguments to hadoop job
Date Tue, 22 Jan 2013 02:27:13 GMT
Just add
System.out.println (baseCount);
To check if expected value is getting displayed. If not then the value is
not set propely in jobconf.

Also you don't need to remove static. Sorry about that, it doesn't change
the expected output, though it's not required.
 On Jan 22, 2013 7:34 AM, "jamal sasha" <jamalshasha@gmail.com> wrote:

> Hi,
>   Thanks for taking out time.
> Will this not work : conf.setInt("basecount",200000); ??
> I am not sure how to add loger or syso (new to both java and hadoop :( )
>
>
>
> On Mon, Jan 21, 2013 at 5:55 PM, Satbeer Lamba <satbeer.lamba@gmail.com>wrote:
>
>> Please be sure that you are getting the value of baseSum in reducer by
>> adding a logger or syso.
>>
>> Also consider removing static in declaration of baseSum as it would add
>> counts of previous keys.
>> On Jan 22, 2013 7:17 AM, "jamal sasha" <jamalshasha@gmail.com> wrote:
>>
>>> The second one.
>>> If the word hello appears once, its count is  2000001.
>>> :)
>>>
>>>
>>> On Mon, Jan 21, 2013 at 5:40 PM, Satbeer Lamba <satbeer.lamba@gmail.com>wrote:
>>>
>>>> Do you mean to say you want to count the words from 200000 th line
>>>> onwards?
>>>>
>>>> OR
>>>>
>>>> You want to start counting from 2000000?
>>>> For example if HELLO appears once it's count is 2000001.
>>>>
>>>> Please clarify
>>>> On Jan 22, 2013 5:22 AM, "jamal sasha" <jamalshasha@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>   Lets say I have the standard helloworld program
>>>>>
>>>>> http://hadoop.apache.org/docs/r0.17.0/mapred_tutorial.html#Example%3A+WordCount+v2.0
>>>>>
>>>>> Now, lets say, I want to start the counting not from zero but from
>>>>> 200000.
>>>>> So my reference line is 200000.
>>>>>
>>>>> I modified the Reduce code as following:
>>>>>  public static class Reduce extends MapReduceBase implements
>>>>> Reducer<Text, IntWritable, Text, IntWritable> {
>>>>>      *private static int baseSum ;*
>>>>> *      public void configure(JobConf job){*
>>>>> *      baseSum = Integer.parseInt(job.get("basecount"));*
>>>>> *      *
>>>>> *      }*
>>>>>        public void reduce(Text key, Iterator<IntWritable> values,
>>>>> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
>>>>> IOException {
>>>>>          int sum =* baseSum*;
>>>>>         while (values.hasNext()) {
>>>>>           sum += values.next().get();
>>>>>          }
>>>>>         output.collect(key, new IntWritable(sum));
>>>>>       }
>>>>>      }
>>>>>
>>>>>
>>>>> And in main added:
>>>>>    conf.setInt("basecount",200000);
>>>>>
>>>>>
>>>>>
>>>>> So my hope was this should have done the trick..
>>>>> But its not working. the code is running normally :(
>>>>> How do i resolve this?
>>>>> Thanks
>>>>>
>>>>
>>>
>

Mime
View raw message