hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ninad Raut <hbase.user.ni...@gmail.com>
Subject Re: Why only few map tasks are running at a time inspite of plenty of scope for remaining?
Date Fri, 24 Jul 2009 10:32:07 GMT
If your data is stored just on one regionserver you will have only one map
> inspite of setting
>  conf.set("mapred.tasktracker.map.tasks.maximum", "2");
> there are two approaches:
>
> 1) Do a manual table split
>
   2) Write a custom input split which will divide the tables rows amongst
the maps.


> On Fri, Jul 24, 2009 at 4:54 AM, akhil1988 <akhilanger@gmail.com> wrote:
>
>>
>> Hi all,
>>
>> I am using a HTable as input to my map jobs and my reducer outputs to
>> another Htable. There are 10 regions of my input HTable. And I have set
>>        conf.set("mapred.tasktracker.map.tasks.maximum", "2");
>>        conf.set("mapred.tasktracker.map.tasks.maximum", "2");
>>       c.setNumReduceTasks(26);
>> My cluster contains 15 nodes(out of which 2 are maters). When I run my
>> job,
>> only 2 map tasks run at a time and the remaining 8 are shown as pending.
>> 24
>> reduce tasks(out of 26) also get started initially and remaing 2 are shown
>> as pending. I am confused why only 2 tasks are running at a time, though
>> there are a total of 26 slots for map tasks.
>>
>> However, this does not happen when I run jobs in which I take files as
>> inputs(i.e. only simple MapReduce jobs and not involving HBase at all).
>> Only
>> when a Htable is taken as input very few map tasks run concurrently than
>> expected.
>>
>> Can anyone suggest why this is happening?
>>
>> What I have observed in simple mapreduce jobs that first all map tasks are
>> instantiated and then reduce tasks. But  this does not seem to be
>> happening
>> in HTable case??
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Why-only-few-map-tasks-are-running-at-a-time-inspite-of-plenty-of-scope-for-remaining--tp24636315p24636315.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message