hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16032) Possible memory leak in StoreScanner
Date Thu, 16 Jun 2016 15:32:06 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15334000#comment-15334000
] 

Hadoop QA commented on HBASE-16032:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} |
{color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 57s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s {color} |
{color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s {color} |
{color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s {color} |
{color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s {color} |
{color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s {color} |
{color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s {color} |
{color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 29m 5s {color}
| {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2
2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 32s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s {color} |
{color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s {color} |
{color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 43s {color} | {color:green}
hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 56s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12810997/HBASE-16032_v3.patch
|
| JIRA Issue | HBASE-16032 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 6d02f36 |
| Default Java | 1.7.0_80 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 /home/jenkins/jenkins-slave/tools/hudson.model.JDK/JDK_1.7_latest_:1.7.0_80
|
| findbugs | v3.0.0 |
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/2256/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/2256/console |
| Powered by | Apache Yetus 0.2.1   http://yetus.apache.org |


This message was automatically generated.



> Possible memory leak in StoreScanner
> ------------------------------------
>
>                 Key: HBASE-16032
>                 URL: https://issues.apache.org/jira/browse/HBASE-16032
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 1.2.1, 1.1.5, 0.98.20
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0.0, 1.1.6, 1.3.1, 0.98.21, 1.2.3
>
>         Attachments: HBASE-16032.patch, HBASE-16032_v2.patch, HBASE-16032_v3.patch
>
>
> We observed frequent fullGC of RS in our production environment, and after analyzing
the heapdump, we found large memory occupancy by HStore#changedReaderObservers, the map is
surprisingly containing 7500w objects...
> After some debugging, I located some possible memory leak in StoreScanner constructor:
> {code}
>   public StoreScanner(Store store, ScanInfo scanInfo, Scan scan, final NavigableSet<byte[]>
columns,
>       long readPt)
>   throws IOException {
>     this(store, scan, scanInfo, columns, readPt, scan.getCacheBlocks());
>     if (columns != null && scan.isRaw()) {
>       throw new DoNotRetryIOException("Cannot specify any column for a raw scan");
>     }
>     matcher = new ScanQueryMatcher(scan, scanInfo, columns,
>         ScanType.USER_SCAN, Long.MAX_VALUE, HConstants.LATEST_TIMESTAMP,
>         oldestUnexpiredTS, now, store.getCoprocessorHost());
>     this.store.addChangedReaderObserver(this);
>     // Pass columns to try to filter out unnecessary StoreFiles.
>     List<KeyValueScanner> scanners = getScannersNoCompaction();
>     ...
>     seekScanners(scanners, matcher.getStartKey(), explicitColumnQuery
>         && lazySeekEnabledGlobally, parallelSeekEnabled);
>     ...
>     resetKVHeap(scanners, store.getComparator());
>   }
> {code}
> If there's any Exception thrown after {{this.store.addChangedReaderObserver(this)}},
the returned scanner might be null and there's no chance to remove the scanner from changedReaderObservers,
like in {{HRegion#get}}
> {code}
>     RegionScanner scanner = null;
>     try {
>       scanner = getScanner(scan);
>       scanner.next(results);
>     } finally {
>       if (scanner != null)
>         scanner.close();
>     }
> {code}
> What's more, all exception thrown in the {{HRegion#getScanner}} path will cause scanner==null
then memory leak, so we also need to handle this part.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message