Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E69217E19 for ; Sat, 11 Oct 2014 03:00:09 +0000 (UTC) Received: (qmail 16202 invoked by uid 500); 11 Oct 2014 03:00:07 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 16133 invoked by uid 500); 11 Oct 2014 03:00:07 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 16117 invoked by uid 99); 11 Oct 2014 03:00:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Oct 2014 03:00:06 +0000 X-ASF-Spam-Status: No, hits=1.0 required=5.0 tests=FREEMAIL_REPLY,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kdiaa@hotmail.com designates 157.55.1.141 as permitted sender) Received: from [157.55.1.141] (HELO DUB004-OMC2S2.hotmail.com) (157.55.1.141) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 11 Oct 2014 03:00:01 +0000 Received: from DUB128-W94 ([157.55.1.138]) by DUB004-OMC2S2.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.22751); Fri, 10 Oct 2014 19:59:40 -0700 X-TMN: [AGQQ1W4WEfmzHpjUsjmuAvoHyLA1IIkq] X-Originating-Email: [kdiaa@hotmail.com] Message-ID: From: Khaled Elmeleegy To: "user@hbase.apache.org" Subject: RE: HBase read performance Date: Fri, 10 Oct 2014 19:59:39 -0700 Importance: Normal In-Reply-To: References: <1412268037.52131.YahooMailNeo@web140603.mail.bf1.yahoo.com>, , <1412273759.20803.YahooMailNeo@web140604.mail.bf1.yahoo.com>, <1412284847.68478.YahooMailNeo@web140601.mail.bf1.yahoo.com>, <1412293938.78812.YahooMailNeo@web140606.mail.bf1.yahoo.com>,,, ,,, ,, ,, Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 11 Oct 2014 02:59:40.0220 (UTC) FILETIME=[665D07C0:01CFE4FF] X-Virus-Checked: Checked by ClamAV on apache.org Nick=2C=0A= =0A= I am still prototyping=2C hence the small instance. I thought=A0XX:MaxDirec= tMemorySize is subset of=A0Xmx=2C so I thought the total is 6GB. You are sa= ying they are added. If so=2C that's not good for performance as the JVM wi= ll swap. However=2C this can't be the reason for the bug.=A0=0A= =0A= For now=2C I guess the only option is to use=A0LruBlockCache. However=2C I = expect it to suffer from significant GC=2C as I churn memory fast because I= want to cache blocks on writes and I do significant writing.=0A= =0A= Khaled=0A= =0A= ----------------------------------------=0A= > From: ndimiduk@gmail.com=0A= > Date: Fri=2C 10 Oct 2014 15:00:45 -0700=0A= > Subject: Re: HBase read performance=0A= > To: user@hbase.apache.org=0A= >=0A= > Hang on=2C how are you using 11G total memory? m1.large only has 7.5G tot= al=0A= > RAM.=0A= >=0A= > On Fri=2C Oct 10=2C 2014 at 2:56 PM=2C Nick Dimiduk = wrote:=0A= >=0A= >> ByteBuffer position math errors makes me suspect #1 cacheonwrite and #2= =0A= >> bucketcache (and #3 their use in combination =3B) )=0A= >>=0A= >> 11G memory is a small enough footprint that I'd not bother with=0A= >> BucketCache=3B just stay on heap with default LruBlockCache.=0A= >>=0A= >> On Fri=2C Oct 10=2C 2014 at 2:11 PM=2C Stack wrote:= =0A= >>=0A= >>> On Fri=2C Oct 10=2C 2014 at 10:59 AM=2C Khaled Elmeleegy =0A= >>> wrote:=0A= >>>=0A= >>>> Yes=2C I can reproduce it with some work.=0A= >>>> The workload is basically as follows:=0A= >>>> There are writers streaming writes to a table. Then=2C there is a read= er=0A= >>>> (invoked via a web interface). The reader does a 1000 parallel reverse= =0A= >>>> scans=2C all end up hitting the same region in my case. The scans are= =0A= >>>> effectively "gets" as I just need to get one record off of each of=0A= >>> them. I=0A= >>>> just need to do a "reverse" get=2C which is not supported (would be gr= eat=0A= >>> to=0A= >>>> have :))=2C so I do it via reverse scan. After few tries=2C the reader= =0A= >>>> consistently hits this bug.=0A= >>>>=0A= >>>> This happens with these config changes:=0A= >>>> hbase-env:HBASE_REGIONSERVER_OPTS=3D-Xmx6G -XX:MaxDirectMemorySize=3D5= G=0A= >>>> -XX:CMSInitiatingOccupancyFraction=3D88 -XX:+AggressiveOpts -verbose:g= c=0A= >>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog=0A= >>>> gc:/tmp/hbase-regionserver-gc.loghbase-site:=0A= >>>> hbase.bucketcache.ioengine=3Doffheap=0A= >>>> hbase.bucketcache.size=3D4196=0A= >>>> hbase.rs.cacheblocksonwrite=3Dtrue=0A= >>>> hfile.block.index.cacheonwrite=3Dtrue=0A= >>>> hfile.block.bloom.cacheonwrite=3Dtrue=0A= >>>>=0A= >>>> Interestingly=2C without these config changes=2C I can't reproduce the= =0A= >>> problem.=0A= >>>=0A= >>>=0A= >>>=0A= >>> How hard to play w/ combinations? Could you eliminate the cacheonwrites= =0A= >>> on=0A= >>> one server and see if that cures the issue? Could trun off block cache = on=0A= >>> another to see if that the problem? Anything in your .out files related= ?=0A= >>>=0A= >>> St.Ack=0A= >>>=0A= >>=0A= >>=0A= =