Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 30 May 2014 17:09:02 +0000 (UTC)
From: "Andrew Purtell (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12717047.1401285216624.48520.1401469742386@arcas>
In-Reply-To: <JIRA.12717047.1401285216624@arcas>
References: <JIRA.12717047.1401285216624@arcas>
Subject: [jira] [Commented] (HBASE-11263) Share the open/close store file
 thread pool for all store in a region
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HBASE-11263?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D140=
13948#comment-14013948 ]=20

Andrew Purtell commented on HBASE-11263:
----------------------------------------

Do we need storeFileThreadPoolCounter? Looks like this is being used effect=
ively as a boolean flag when you could achieve the same by testing if store=
FileThreadPool is null or not in an exclusive section? Save some heap per H=
Region.

What happens if you use one thread pool for opening and closing store files=
 per RegionServer? If we have thousands of HRegions and they all have threa=
d pools, surely we are not able to fully schedule that degree of parallelis=
m?

> Share the open/close store file thread pool for all store in a region
> ---------------------------------------------------------------------
>
>                 Key: HBASE-11263
>                 URL: https://issues.apache.org/jira/browse/HBASE-11263
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>    Affects Versions: 0.99.0
>            Reporter: Liu Shaohui
>            Assignee: Liu Shaohui
>            Priority: Minor
>         Attachments: HBASE-11263-trunk-v1.diff
>
>
> Currently, the open/close store file thread pool is divided equally to al=
l stores of a region.=20
> {code}
>   protected ThreadPoolExecutor getStoreFileOpenAndCloseThreadPool(
>       final String threadNamePrefix) {
>     int numStores =3D Math.max(1, this.htableDescriptor.getFamilies().siz=
e());
>     int maxThreads =3D Math.max(1,
>         conf.getInt(HConstants.HSTORE_OPEN_AND_CLOSE_THREADS_MAX,
>             HConstants.DEFAULT_HSTORE_OPEN_AND_CLOSE_THREADS_MAX)
>             / numStores);
>     return getOpenAndCloseThreadPool(maxThreads, threadNamePrefix);
>   }
> {code}
> This is not very optimal in following scenarios:
> # The data of some column families are very large and there are many hfil=
es in those stores, and others may be very small and in-memory column famil=
ies.=20
> # Usually we preserve some column families for later needs. The thread po=
ol for these column families are wasted=E3=80=82
> The simple way is to share a big thread pool for all stores to open/close=
 hfiles. =20
> Suggestions are welcomed.=20
> Thanks.=20


--
This message was sent by Atlassian JIRA
(v6.2#6252)