hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars George (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HBASE-1287) Partitioner class not used in TableMapReduceUtil.initTableReduceJob()
Date Wed, 25 Mar 2009 12:14:51 GMT

     [ https://issues.apache.org/jira/browse/HBASE-1287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars George updated HBASE-1287:
-------------------------------

    Description: 
Upon checking the available utility methods in TableMapReduceUtil I came across this code

{code}
  public static void initTableReduceJob(String table,
    Class<? extends TableReduce> reducer, JobConf job, Class partitioner)
  throws IOException {
    job.setOutputFormat(TableOutputFormat.class);
    job.setReducerClass(reducer);
    job.set(TableOutputFormat.OUTPUT_TABLE, table);
    job.setOutputKeyClass(ImmutableBytesWritable.class);
    job.setOutputValueClass(BatchUpdate.class);
    if (partitioner != null) {
      job.setPartitionerClass(HRegionPartitioner.class);
      HTable outputTable = new HTable(new HBaseConfiguration(job), table);
      int regions = outputTable.getRegionsInfo().size();
      if (job.getNumReduceTasks() > regions){
    	job.setNumReduceTasks(outputTable.getRegionsInfo().size());
      }
    }
  }
{code}

It seems though as it should be

{code}
    if (partitioner != null) {
      job.setPartitionerClass(partitioner);
{code}

and the provided HRegionPartitioner can be handed in to that call or a custom one can be provided.

  was:
Upon checking the available utility methods in TableMapReduceUtil I came across this code:

{code}
  public static void initTableReduceJob(String table,
    Class<? extends TableReduce> reducer, JobConf job, Class partitioner)
  throws IOException {
    job.setOutputFormat(TableOutputFormat.class);
    job.setReducerClass(reducer);
    job.set(TableOutputFormat.OUTPUT_TABLE, table);
    job.setOutputKeyClass(ImmutableBytesWritable.class);
    job.setOutputValueClass(BatchUpdate.class);
    if (partitioner != null) {
      job.setPartitionerClass(HRegionPartitioner.class);
      HTable outputTable = new HTable(new HBaseConfiguration(job), table);
      int regions = outputTable.getRegionsInfo().size();
      if (job.getNumReduceTasks() > regions){
    	job.setNumReduceTasks(outputTable.getRegionsInfo().size());
      }
    }
  }
{code}

It seems though as it should be

{code}
    if (partitioner != null) {
      job.setPartitionerClass(partitioner);
{code}

and the provided HRegionPartitioner can be handed in to that call or a custom one can be provided.


> Partitioner class not used in TableMapReduceUtil.initTableReduceJob()
> ---------------------------------------------------------------------
>
>                 Key: HBASE-1287
>                 URL: https://issues.apache.org/jira/browse/HBASE-1287
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Lars George
>            Assignee: Lars George
>
> Upon checking the available utility methods in TableMapReduceUtil I came across this
code
> {code}
>   public static void initTableReduceJob(String table,
>     Class<? extends TableReduce> reducer, JobConf job, Class partitioner)
>   throws IOException {
>     job.setOutputFormat(TableOutputFormat.class);
>     job.setReducerClass(reducer);
>     job.set(TableOutputFormat.OUTPUT_TABLE, table);
>     job.setOutputKeyClass(ImmutableBytesWritable.class);
>     job.setOutputValueClass(BatchUpdate.class);
>     if (partitioner != null) {
>       job.setPartitionerClass(HRegionPartitioner.class);
>       HTable outputTable = new HTable(new HBaseConfiguration(job), table);
>       int regions = outputTable.getRegionsInfo().size();
>       if (job.getNumReduceTasks() > regions){
>     	job.setNumReduceTasks(outputTable.getRegionsInfo().size());
>       }
>     }
>   }
> {code}
> It seems though as it should be
> {code}
>     if (partitioner != null) {
>       job.setPartitionerClass(partitioner);
> {code}
> and the provided HRegionPartitioner can be handed in to that call or a custom one can
be provided.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message