hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sriram <rsriram...@gmail.com>
Subject Re: Reducer
Date Mon, 12 Sep 2011 12:50:53 GMT
 
> - Could you detail your job out? Does it use TableMapReduceUtil? What
> does it do and what all of HBase API does it call after you set your
> number of reduce tasks into the job configuration? (Note: If this is a
> general MapReduce query, you should send it to
> mapreduce-user@... instead)
 
Yes i am using TableMapReduceUtil.

> - You, or your submission code may be calling
> TableMapReduceUtil.limitNumReduceTasks(…) or
> TableMapReduceUtil.setNumReduceTasks(…) both of which reset the no. of
> reducers based on the number of output table regions at max. In this
> case, its better to see if you have only a single large region in your
> table, and get to fixing/splitting that (as it wouldn't parallelize
> otherwise).

TableMapReduceUtil.initTableReducerJob("tablename",IntSumReducer.class,job);
TableMapReduceUtil.limitNumReduceTasks("tablename",job); 

I am using only one table 
 
> - I believe a single reducer applied over a large data would cause a
> lot of time being spent in sorting, which is possibly why you're
> noticing the delay from 33-66 progress %.

yes i figured out that.Is there a way to increase the reducers....???





Mime
View raw message