hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Zaliva <kroko...@gmail.com>
Subject Re: unable to figure out this exception from reduce task
Date Wed, 16 Jan 2008 06:58:21 GMT
On Jan 15, 2008, at 22:09, Jim the Standing Bear wrote:

Only thing I noticed (compared to my code) is missing:

   client.setConf(conf);

before client.run(conf).

In this case default input format is used which uses  
LongWriteableComparable as
a key. Not sure if this is a case, but something worth checking.

Vadim


> Well, I also wish it was this simple, but as I said in the original
> message, I never wanted to use LongWritable at all.  Here is how I set
> the job conf, and after that, is the reduce task.  Also, if I got the
> incorrect output key/value type, shouldn't it always fail as soon as
> the reduce task is run?  But my code behaves strangely that sometimes
> the exception didn't get thrown until a few iterations had been
> successfully passed...  Does the code reveal something that I missed?
> Thanks.
>
>            JobConf countNewCatalogJobConf = new
> JobConf(ThreddsCatalogIndexer.class);
>            countNewCatalogJobConf.setJobName("Count-New-Catalog-" +  
> iteration);
>            countNewCatalogJobConf.setInputPath(newCatUrlDir);
>             
> countNewCatalogJobConf.setInputFormat(KeyValueTextInputFormat.class);
>            countNewCatalogJobConf.setOutputPath(newCatalogCountDir);
>            countNewCatalogJobConf.setOutputKeyClass(Text.class);
>            countNewCatalogJobConf.setOutputValueClass(Text.class);
>             
> countNewCatalogJobConf.setReducerClass(NewCatalogCounter.class);
>            countNewCatalogJobConf.setNumReduceTasks(1);
>            JobClient.runJob(countNewCatalogJobConf);
>
>
>
>
>    public void reduce(WritableComparable key, Iterator values,
> OutputCollector output, Reporter reporter) throws IOException {
>        long sum = 0;
>        if (key.toString().equals("NEWCAT")) {
>            while (values.hasNext()) {
>                sum++;
>            }
>        }
>        Text sumText = new Text();
>        String sumString = (new Long(sum)).toString();
>        sumText.set(sumString);
>        output.collect(key, sumText);
>    }
>
>

Mime
View raw message