hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bertrand Dechoux <decho...@gmail.com>
Subject Re: why reduce doesn't work by HBase?
Date Thu, 15 Nov 2012 20:56:51 GMT
Hi Yong,

A few ways to understand a problem (with Hadoop) :
1) Write tests, using MRUnit for example (http://mrunit.apache.org/)
2) Use @Override to make sure you are indeed overriding a method
3) Use a break point while debugging

The answer to your current problem : o.a.h.mapreduce.Reducer method has no
Iterator parameter but it does have a Iterable parameter...

Regards

Bertrand

PS : It is absolutely not related to HBase.

On Thu, Nov 15, 2012 at 8:42 PM, yonghu <yongyong313@gmail.com> wrote:

> Dear all,
>
> I use HBase as data source and hdfs as data sink. I wrote the program
> which calculate the versions for each cell as follows:
>
> public class ReturnMore { //only Map side has no problem, has already
> returned 3 versions
>
>    public static class Map extends TableMapper<Text, Text>{
>       private Text outkey = new Text();
>       private Text outval = new Text();
>       public void map(ImmutableBytesWritable key, Result values, Context
> context){
>          KeyValue[] kvs = values.raw();
>          String row_key;
>          String col_fam;
>          String col;
>          String val;
>          String finalKey;
>          String finalValue;
>          for(KeyValue kv : kvs){
>             row_key = Bytes.toString(kv.getRow());
>             col_fam = Bytes.toString(kv.getFamily());
>             col = Bytes.toString(kv.getQualifier());
>             val = Bytes.toString(kv.getValue());
>             finalKey = row_key + "/" + col_fam + "/" + col;
>             finalValue = new Long(kv.getTimestamp()).toString() + "/" +
> val;
>             outkey.set(finalKey);
>             outval.set(finalValue);
>             try {
>                context.write(outkey, outval);
>             } catch (IOException e) {
>                e.printStackTrace();
>             } catch (InterruptedException e) {
>                e.printStackTrace();
>             }
>          }
>       }
>    }
>
>    public static class Reduce extends Reducer<Text, Text, Text,
> IntWritable>{ //the desired output is key + 3
>       private IntWritable occ = new IntWritable();
>       public void reduce(Text key, Iterator<Text> values, Context
> context){
>          int i = 0;
>          while(values.hasNext()){
>             i++;
>          }
>          occ.set(i);
>          try {
>             context.write(key, occ);
>          } catch (IOException e) {
>             e.printStackTrace();
>          } catch (InterruptedException e) {
>             e.printStackTrace();
>          }
>       }
>    }
>    public static void main(String[] args) throws Exception{
>       Configuration conf = HBaseConfiguration.create();
>       Job job = new Job(conf);
>       job.setJarByClass(ReturnMore.class);
>       Scan scan = new Scan();
>       scan.setMaxVersions();
>       job.setReducerClass(Reduce.class);
>       job.setOutputKeyClass(Text.class);
>       job.setOutputValueClass(IntWritable.class);
>       job.setNumReduceTasks(1);
>       TableMapReduceUtil.initTableMapperJob("test", scan, Map.class,
> Text.class, Text.class, job);
>       FileSystem file = FileSystem.get(conf);
>       Path path = new Path("/hans/test/");
>       if(file.exists(path)){
>          file.delete(path,true);
>       }
>       FileOutputFormat.setOutputPath(job, path);
>       System.exit(job.waitForCompletion(true)?0:1);
>    }
> }
>
> I tested this code in both hbase 0.92.1 and 0.94. However, if I run
> this code, it always outputs the content for each cell not as the
> output as I defined in reduce function (key + occurrence for each
> cell). Can anyone give me advices? By the way, I run it on
> pseudo-mode.
>
> regards!
>
> Yong
>



-- 
Bertrand Dechoux

Mime
View raw message