Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 15 Jun 2016 14:59:09 +0000 (UTC)
From: "Yu Li (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12979387.1466002256000.5606.1466002749397@Atlassian.JIRA>
In-Reply-To: <JIRA.12979387.1466002256000@Atlassian.JIRA>
References: <JIRA.12979387.1466002256000@Atlassian.JIRA> <JIRA.12979387.1466002256082@arcas>
Subject: [jira] [Updated] (HBASE-16032) Possible memory leak in StoreScanner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 15 Jun 2016 14:59:11 -0000


     [ https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yu Li updated HBASE-16032:
--------------------------
        Fix Version/s: 0.98.21
                       1.1.6
                       1.2.2
                       1.3.0
                       2.0.0
    Affects Version/s: 1.2.1
                       1.1.5
                       0.98.20
               Status: Patch Available  (was: Open)

Submit patch for HadoopQA

> Possible memory leak in StoreScanner
> ------------------------------------
>
>                 Key: HBASE-16032
>                 URL: https://issues.apache.org/jira/browse/HBASE-16032
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.20, 1.1.5, 1.2.1
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0.0, 1.3.0, 1.2.2, 1.1.6, 0.98.21
>
>         Attachments: HBASE-16032.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after analyzing the heapdump, we found large memory occupancy by HStore#changedReaderObservers, the map is surprisingly containing 7500w objects...
> After some debugging, I located some possible memory leak in StoreScanner constructor:
> {code}
>   public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final NavigableSet<byte[]> columns,
>       long readPt)
>   throws IOException {
>     this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
>     if (columns != null && scan.isRaw()) {
>       throw new DoNotRetryIOException("Cannot specify any column for a raw scan");
>     }
>     matcher = new ScanQueryMatcher(scan, scanInfo, columns,
>         ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
>         oldestUnexpiredTS, now, store.getCoprocessorHost());
>     this.store.addChangedReaderObserver(this);
>     // Pass columns to try to filter out unnecessary StoreFiles.
>     List<KeyValueScanner> scanners = getScannersNoCompaction();
>     ...
>     seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
>         && lazySeekEnabledGlobally, parallelSeekEnabled);
>     ...
>     resetKVHeap(scanners, store.getComparator());
>   }
> {code}
> If there's any Exception thrown after {{this.store.addChangedReaderObserver(this)}}, the returned scanner might be null and there's no chance to remove the scanner from changedReaderObservers, like in HRegion#get
> {code}
>     RegionScanner scanner = null;
>     try {
>       scanner = getScanner(scan);
>       scanner.next(results);
>     } finally {
>       if (scanner != null)
>         scanner.close();
>     }
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)