From issues-return-370407-archive-asf-public=cust-asf.ponee.io@hbase.apache.org Wed Dec 5 14:32:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id C9B7A18067A for ; Wed, 5 Dec 2018 14:32:05 +0100 (CET) Received: (qmail 65855 invoked by uid 500); 5 Dec 2018 13:32:04 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 65837 invoked by uid 99); 5 Dec 2018 13:32:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Dec 2018 13:32:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 52A14182834 for ; Wed, 5 Dec 2018 13:32:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.301 X-Spam-Level: X-Spam-Status: No, score=-110.301 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id h4H3CZKmxk51 for ; Wed, 5 Dec 2018 13:32:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 11D6660EDD for ; Wed, 5 Dec 2018 13:32:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 02D83E0E5D for ; Wed, 5 Dec 2018 13:32:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 9373024DDA for ; Wed, 5 Dec 2018 13:32:00 +0000 (UTC) Date: Wed, 5 Dec 2018 13:32:00 +0000 (UTC) From: "Zheng Hu (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-21551) Memory leak when use scan with STREAM at server side MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-21551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21551: ----------------------------- Description: We open the RegionServerScanner with STREAM as following: {code} RegionScannerImpl#initializeScanners |---> HStore#getScanner |----------> StoreScanner() |-------> StoreFileScanner#getScannersForStoreFiles |------> HStoreFile#getStreamScanner #1 {code} In #1, we put the StoreFileReader into a concurrent hash map streamReaders, but not remove the StreamReader from streamReaders until closing the store file. So if we scan with stream with so many times, the streamReaders hash map will be exploded. we can see the heap dump in the attached heap-dump.jpg. I found this bug, because when i benchmark the scan performance by using YCSB in a cluster (heap size of RS is 50g), the Rs was easy to occur a long time full gc ( ~ 110 sec).... was: We open the RegionServerScanner with STREAM as following: {code} RegionScannerImpl#initializeScanners |---> HStore#getScanner |----------> StoreScanner() |-------> StoreFileScanner#getScannersForStoreFiles |------> HStoreFile#getStreamScanner #1 {code} In #1, we put the StoreFileReader into a concurrent hash map streamReaders, but not remove the StreamReader from streamReaders until closing the store file. So if we scan with stream with so many times, the streamReaders hash map will be exploded. we can see the heap dump in the attached heap-dump.jpg. I found this bug, because when i benchmark the scan performance by using YCSB in a cluster (heap size of RS is 50g), the Rs will be easy to happen a long time full gc ( ~ 110 sec).... > Memory leak when use scan with STREAM at server side > ---------------------------------------------------- > > Key: HBASE-21551 > URL: https://issues.apache.org/jira/browse/HBASE-21551 > Project: HBase > Issue Type: Bug > Components: regionserver > Reporter: Zheng Hu > Assignee: Zheng Hu > Priority: Blocker > Fix For: 3.0.0, 2.2.0, 2.1.2, 2.0.4 > > Attachments: heap-dump.jpg > > > We open the RegionServerScanner with STREAM as following: > {code} > RegionScannerImpl#initializeScanners > |---> HStore#getScanner > |----------> StoreScanner() > |-------> StoreFileScanner#getScannersForStoreFiles > |------> HStoreFile#getStreamScanner #1 > {code} > In #1, we put the StoreFileReader into a concurrent hash map streamReaders, but not remove the StreamReader from streamReaders until closing the store file. > So if we scan with stream with so many times, the streamReaders hash map will be exploded. we can see the heap dump in the attached heap-dump.jpg. > I found this bug, because when i benchmark the scan performance by using YCSB in a cluster (heap size of RS is 50g), the Rs was easy to occur a long time full gc ( ~ 110 sec).... -- This message was sent by Atlassian JIRA (v7.6.3#76005)