Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B5EFBC4C8 for ; Tue, 23 Dec 2014 15:01:13 +0000 (UTC) Received: (qmail 8554 invoked by uid 500); 23 Dec 2014 15:01:13 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 8503 invoked by uid 500); 23 Dec 2014 15:01:13 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 8491 invoked by uid 99); 23 Dec 2014 15:01:13 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Dec 2014 15:01:13 +0000 Date: Tue, 23 Dec 2014 15:01:13 +0000 (UTC) From: "Jurriaan Mous (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-12736) Let MetaScanner recycle a given Connection MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-12736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12736: ---------------------------------- Attachment: HBASE-12736-v4.patch You are right. It uses way less Threads in general. I was too focussed on the graph. I was worried about the more active threads and more memory usage. But it gets all cleaned up and they end up the same. Looking into what was causing it it was pretty simple. Since all MetaScanners use the same connection, they use the same thread pool. And now each scanner does not close the connection and thus they don't clean up their threads along the way. And the thread pool limit is not reached while scanning so the peaks are higher. > Let MetaScanner recycle a given Connection > ------------------------------------------ > > Key: HBASE-12736 > URL: https://issues.apache.org/jira/browse/HBASE-12736 > Project: HBase > Issue Type: Improvement > Components: Client > Reporter: Jurriaan Mous > Assignee: Jurriaan Mous > Attachments: HBASE-12736-v1.patch, HBASE-12736-v2.patch, HBASE-12736-v3.patch, HBASE-12736-v4.patch, HBASE-12736.patch, profile_TestMetaScanner_after_patch.png, profile_TestMetaScanner_before_patch.png > > > It is very heavy to create a Connection on each meta scan. Connections create RpcClients, RpcClients create RPC connections and all cannot be recycled. > Tests with a lot of metascans are very heavy with the async client. > This issue is to make anything that uses metaScans reuse the same connection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)