Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9C11B200C30 for ; Tue, 21 Feb 2017 04:36:48 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9AC18160B78; Tue, 21 Feb 2017 03:36:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id EB5E8160B73 for ; Tue, 21 Feb 2017 04:36:47 +0100 (CET) Received: (qmail 336 invoked by uid 500); 21 Feb 2017 03:36:47 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 324 invoked by uid 99); 21 Feb 2017 03:36:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Feb 2017 03:36:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 76A1FC1F03 for ; Tue, 21 Feb 2017 03:36:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.999 X-Spam-Level: X-Spam-Status: No, score=0.999 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id xqUNtO_XisiN for ; Tue, 21 Feb 2017 03:36:45 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 289265F5F7 for ; Tue, 21 Feb 2017 03:36:45 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 5F6E1E040F for ; Tue, 21 Feb 2017 03:36:44 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1B8C32411F for ; Tue, 21 Feb 2017 03:36:44 +0000 (UTC) Date: Tue, 21 Feb 2017 03:36:44 +0000 (UTC) From: "Bingbing Wang (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-17671) HBase Thrift2 OutOfMemory MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Tue, 21 Feb 2017 03:36:48 -0000 [ https://issues.apache.org/jira/browse/HBASE-17671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bingbing Wang updated HBASE-17671: ---------------------------------- Attachment: log_gc.log.0.zip Thrift2 GC log > HBase Thrift2 OutOfMemory > ------------------------- > > Key: HBASE-17671 > URL: https://issues.apache.org/jira/browse/HBASE-17671 > Project: HBase > Issue Type: Bug > Components: Thrift > Affects Versions: 0.98.6 > Environment: Product > Reporter: Bingbing Wang > Priority: Critical > Attachments: hbase-site.xml, log_gc.log.0.zip > > > We have a HBase Thrift2 server deployed on Windows, basically the physical view looks like: > QueryEngine <==> HBase Thrift2 <==> HBase cluster > Here QueryEngine is a C++ application, and HBase cluster is a about 50-nodes HBase cluster (CDH 5.3.3, namely Hbase version 0.98.6). > Our Thrift2 Java options looks like: > -server -Xms4096m -Xmx4096m -XX:MaxDirectMemorySize=8192m -XX:+HeapDumpOnOutOfMemoryError -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=4M -XX:InitiatingHeapOccupancyPercent=40 -XX:+PrintAdaptiveSizePolicy -XX:+PrintPromotionFailure -Dhbase.log.dir=d:\vhayu\thrift2\log -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:PrintFLSStatistics=1 -Xloggc:log_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=200M -Dhbase.log.file=hbase-thrift2.log -Dhbase.home.dir=D:\vhayu\thrift2\hbase0.98 -Dhbase.id.str=root -Dlog4j.info -Dhbase.root.logger=INFO,DRFA -cp "d:\vhayu\thrift2\hbase0.98\*;d:\vhayu\thrift2\conf" org.apache.hadoop.hbase.thrift2.ThriftServer -b 127.0.0.1 -f framed start > The phenomenon of the issue is that after some time running, Thrift2 sometimes reports OOM and heap dump file (.hprof) file was generated. The consequence of this will always trigger high latency form HBase cluster. -- This message was sent by Atlassian JIRA (v6.3.15#6346)