Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of shahab.yunus@gmail.com
 designates 209.85.216.48 as permitted sender)
MIME-Version: 1.0
Date: Thu, 14 Aug 2014 08:23:16 -0400
Message-ID: 
 <CAEo-6+SLTGCQkw58EgdLLN982qsekEPk1RXyZedBib_EPi2NPw@mail.gmail.com>
Subject: Relationship between number of reducers and number of regions in the
 table
From: Shahab Yunus <shahab.yunus@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>,
	"user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a113a666425d307050095f8d7

--001a113a666425d307050095f8d7
Content-Type: text/plain; charset=UTF-8

I couldn't decide that whether it is an HBase question or Hadoop/Yarn.

In the utility class for MR jobs integerated with HBase,
*org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil, *

in the method:


*public static void initTableReducerJob(String table,*
*    Class<? extends TableReducer> reducer, Job job,*
*    Class partitioner, String quorumAddress, String serverClass,*
*    String serverImpl, boolean addDependencyJars) throws IOException;*

Im the above method the following check is added, while setting the number
of reducers that:


*...*
*int regions = outputTable.getRegionsInfo().size();*
*...*
*if (job.getNumReduceTasks() > regions) {*
*        job.setNumReduceTasks(outputTable.getRegionsInfo().size());*
*      }*
*...      *

What is the reason for doing this? And what are the negative effects we
don't follow this? I can think one that, in case of more than one reducer
writing/reading a same region can cause hot-spotting and performance
issues. Are there any other reasons to add this check as well?

Thanks a lot.

Regards,
Shahab

--001a113a666425d307050095f8d7
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I couldn&#39;t decide that whether it is an HBase question=
 or Hadoop/Yarn.<div><br></div><div>In the utility class for MR jobs intege=
rated with HBase, <b>org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil,=
=C2=A0</b></div>
<div><br></div><div>in the method:</div><div><br></div><div><div><br></div>=
<div><b>public static void initTableReducerJob(String table,</b></div><div>=
<b>=C2=A0 =C2=A0 Class&lt;? extends TableReducer&gt; reducer, Job job,</b><=
/div><div>
<b>=C2=A0 =C2=A0 Class partitioner, String quorumAddress, String serverClas=
s,</b></div><div><b>=C2=A0 =C2=A0 String serverImpl, boolean addDependencyJ=
ars) throws IOException;</b></div><div><br></div><div>Im the above method t=
he following check is added, while setting the number of reducers that:</di=
v>
<div><br></div><div><br></div><div><b>...</b></div><div><b>int regions =3D =
outputTable.getRegionsInfo().size();</b><br></div><div><b>...</b></div><div=
><b>if (job.getNumReduceTasks() &gt; regions) {</b></div><div><b>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 job.setNumReduceTasks(outputTable.getRegionsInfo().size()=
);</b></div>
<div><b>=C2=A0 =C2=A0 =C2=A0 }</b></div><div><b>... =C2=A0 =C2=A0 =C2=A0</b=
></div><div>=C2=A0 =C2=A0 =C2=A0=C2=A0</div><div>What is the reason for doi=
ng this? And what are the negative effects we don&#39;t follow this? I can =
think one that, in case of more than one reducer writing/reading a same reg=
ion can cause hot-spotting and performance issues. Are there any other reas=
ons to add this check as well?</div>
</div><div><br></div><div>Thanks a lot.</div><div><br></div><div>Regards,</=
div><div>Shahab</div><div><br></div><div><pre style=3D"color:rgb(0,0,0);wor=
d-wrap:break-word;white-space:pre-wrap"><br></pre><pre style=3D"color:rgb(0=
,0,0);word-wrap:break-word;white-space:pre-wrap">
<br></pre></div></div>

--001a113a666425d307050095f8d7--