hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajeshkumar J <rajeshkumarit8...@gmail.com>
Subject Re: region size for a mapper
Date Mon, 15 May 2017 09:35:31 GMT
Hi,

 Thanks ted. we are using default split policy and our flush size is 64 MB.
And the size is calculated based on the formula

 Math.min(getDesiredMaxFileSize(),initialSize * tableRegionsCount *
tableRegionsCount * tableRegionsCount);

 If this size exceeds max region size (10 GB), then max region size will be
taken or else calculated value will be taken. Suppose if the calculation
returns 11 gb then 10 gb will be considered and it will be sent to mapper.
Default mapper mapred.map.child.java.opts is 1 gb. what my doubt what if
10gb of data is sent to this mapper with 1 gb whether there will be any
issue??

Thanks

On Mon, May 15, 2017 at 2:22 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Split policy may play a role here.
>
> Please take a look at:
> http://hbase.apache.org/book.html#_custom_split_policies
>
> On Mon, May 15, 2017 at 1:48 AM, Rajeshkumar J <
> rajeshkumarit8292@gmail.com>
> wrote:
>
> > Hi,
> >
> >   As we run mapreduce over hbase it will take each region as input for
> each
> > mapper. I have given region max size as 10GB. If i have about 5 gb will
> it
> > take 5 gb of data as input of mappers??
> >
> > Thanks
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message