hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Using Hbase as data sink
Date Mon, 22 Dec 2008 21:43:10 GMT
Jim Twensky wrote:
> Hello Jonathan,
>
> Thanks for the fast response. Yes, my question is on other methods to put
> the same data layout into HBase from my map reduce jobs. I've seen the
> TableOutputFormat but I couldn't find any example usages of it.
>   

Try src/example/mapred.  See SampleUploader.  It does like you want with 
a mapper prepping data and then a reducer to insert into hbase table.  
For how the reduce is configured, see the initTableReduceJob in 
TableMapReduceUtil.  It looks like this:

    job.setOutputFormat(TableOutputFormat.class);
    job.setReducerClass(reducer);
    job.set(TableOutputFormat.OUTPUT_TABLE, table);
    job.setOutputKeyClass(ImmutableBytesWritable.class);
    job.setOutputValueClass(BatchUpdate.class);
    if (partitioner != null) {
      job.setPartitionerClass(HRegionPartitioner.class);
      HTable outputTable = new HTable(new HBaseConfiguration(job), table);
      int regions = outputTable.getRegionsInfo().size();
      if (job.getNumReduceTasks() > regions){
        job.setNumReduceTasks(outputTable.getRegionsInfo().size());
      }
    }

... sets the output format, reducer, output keys, table name, and tries 
to set reduce count.

If any issue with the above, let us know.
St.Ack

Mime
View raw message