hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Number map task and reduce?
Date Mon, 16 Feb 2009 16:36:35 GMT
NguyenHuynh:

The below needs a little qualification in the hbase case (Thanks for the
useful pointers Rasit).  If datanodes (and hbase) are running on same nodes
as TaskTrackers, as is the usual case, then TTs and their hosted maps on
small machines can starve datanodes of cycles.  Suffering datanodes will
reflect in troubled hbase regionserver operation.

For this reason I suggest starting out with a small number of mappers
running in any one TT.

Be sure to checkout the recommendations at the end of this page
http://wiki.apache.org/hadoop/Hbase/Troubleshooting.  Also review the
'Getting Started' (and FAQ), particularly around upping the file descriptor
limits.

St.Ack


On Sun, Feb 15, 2009 at 11:05 PM, Rasit OZDAS <rasitozdas@gmail.com> wrote:

> Hi, NguyenHuynh
>
> If you need a sensitive solution, there is an article giving following
> operations to compute number of maps/reduces:
> Number of maps     :   max(min(block_size, data/#maps), min_split_size)
> Number of reduces :
> 0.95*num_nodes*mapred.tasktracker.reduce.tasks.maximum
>
> If you need an "understandable" solution, try the following link:
> http://wiki.apache.org/hadoop/HowManyMapsAndReduces
>
> If you need just a rough estimation:
> Maps: The right level of parallelism for maps seems to be around
> 10-100 maps per-node.
>
> There is also another calculation (I find it worse than others), I
> couldn't find now. It's something like:
> Maps      : An even number closest to several times bigger than number
> of task trackers.
> Reduces: Same as number of task trackers.
>
> Are you using namenode and jobtracker as tasktrackers, too?
>
> If you have difficulties, please inform..
> Rasit
>
> 2009/2/16 nguyenhuynh <nguyenhuynh@asnet.com.vn>:
> > Hi all!,
> >
> >
> > I have 3 machines use to run Hadoop/hbase map-reduce. I don't known set
> > value for number map tasks and reduces.
> >
> >
> > How many number of task and reduce in this case?
> >
> >
> > Please, help me!
> >
> > Thanks,
> >
> >
> > Regards,
> >
> > NguyenHuynh
> >
> >
> >
>
>
>
> --
> M. Raşit ÖZDAŞ
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message