hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florin P <florinp...@yahoo.com>
Subject RE: Obtain many mappers (or regions)
Date Mon, 27 Jun 2011 14:44:27 GMT
Hello!
  I've found the problem with the number of mappers. We are running the M/R jobs with Oozie
that apparently ignores the set up of the mapred.map.tasks property that is used as a hint
for computing the number of splits. Cite from the TableInputFormatBase#getSplits(old API)
java doc:
"Splits are created in number equal to the smallest between numSplits and
    the number of {@link HRegion}s in the table. If the number of splits is
    smaller than the number of {@link HRegion}s then splits are spanned across
    multiple {@link HRegion}s and are grouped the most evenly possible. In the
    case splits are uneven the bigger splits are placed first in the
    {@link InputSplit} array."

By default the mapred.map.tasks is set up to 2. Applying the above algorithm on my scenario
(and the oozie observation), computing  min(mapred.map.tasks=2,number_of_my_regions=32) then
we obtain the "magic" number of mappers 2. 
  We have observed this behavior, by implementing a Driver for the MR job and setting up the
mapred.map.tasks to 40 let's say. Then the number of mappers are calculated correctly to 32.
  Regards,
   Florin



--- On Mon, 6/27/11, Florin P <florinpico@yahoo.com> wrote:

> From: Florin P <florinpico@yahoo.com>
> Subject: RE: Obtain many mappers (or regions)
> To: user@hbase.apache.org
> Date: Monday, June 27, 2011, 8:46 AM
> Hi!
>   Thank you for your response. As I said, it is a
> temporary table. This table acts as a metadata for long
> tasks processing that we would like to trigger from the
> cluster (as map/reduce jobs) in order that all machines to
> take some of that tasks.
>   I have read the indicated chapter, and then I have
> followed the scenario:
>    1.We have loaded the small data into the
> hbase table
>    2. From the hbase admin interface we
> triggered the split action
>    3. We have seen that 32 new regions were
> created for that table
>    4. We have ran a map/reduce job that
> counts the number of rows
>    5. Only two mappers were created
> What is puzzles me is that only 2 mapper tasks were
> created, even in the indicated book it is stated that
>  (cite)"
> When TableInputFormat, is used to source an HBase table in
> a MapReduce job, its splitter will make a map task for each
> region of the table. Thus, if there are 100 regions in the
> table, there will be 100 map-tasks for the job - regardless
> of how many column families are selected in the Scan.
> "  
> 
> Can you please explain why this is happen? Did we miss some
> property configuration?
> 
> Thank you.
>  regards,
>   Florin
> --- On Mon, 6/27/11, Doug Meil <doug.meil@explorysmedical.com>
> wrote:
> 
> > From: Doug Meil <doug.meil@explorysmedical.com>
> > Subject: RE: Obtain many mappers (or regions)
> > To: "user@hbase.apache.org"
> <user@hbase.apache.org>
> > Date: Monday, June 27, 2011, 8:01 AM
> > Hi there-
> > 
> > If you only have 100 rows I think that HBase might be
> > overkill.
> > 
> > You probably want to start with this to get a
> background on
> > what HBase can do...
> > http://hbase.apache.org/book.html
> > .. there is a section on MapReduce with HBase as
> well.
> > 
> > -----Original Message-----
> > From: Florin P [mailto:florinpico@yahoo.com]
> > 
> > Sent: Monday, June 27, 2011 4:53 AM
> > To: user@hbase.apache.org
> > Subject: Obtain many mappers (or regions)
> > 
> > Hello!
> > I have the following scenario:
> > 1. A temporary HBase table with small number of rows
> (aprox
> > 100) 2. A cluster with 2 machines that I would like
> to
> > crunch the data contained in the rows  
> > 
> > I would like to create two mappers that will crunch
> the
> > data from rows. 
> >  How can I achieve this?
> > A general question is: 
> >   how we can obtain many mappers to crunch small
> data
> > quantity?
> > 
> > Thank you.
> >   Regards,
> >   Florin  
> >
> 

Mime
View raw message