Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=RRv6/2ByIKZKqxIaKydusYkqPKhPMBc7CKtdIhHkuUFnOxzW+P/s5yCHT42G1qleR+94s/rjbXhID/qzp5VtECgRaNddxtVyhUNmv9i45tIzaiaaKTpdIGIH8BVsSw0C1KQoxvflxjJ0ACnjhXaMjm6GzxWr+isRP/KCZpMTHQ8=;
References: 
 <DC5EBE7F3610EB4CA5C7E92D76873E15180DD42ADD@exchange2007.carrieriq.com>
	<DC5EBE7F3610EB4CA5C7E92D76873E15180DD42ADE@exchange2007.carrieriq.com>
	<CALte62weJJV1V-Yjn+YCjiXbZR6Pj8r_Jqn-C-3zg98NdiaJhA@mail.gmail.com>
 <CAAg3a2ryAfSqcGm3VdExB6n-FoRANjXy6ZLqwABZ90ybxBi8jg@mail.gmail.com>
 <1381541319.43070.YahooMailNeo@web140603.mail.bf1.yahoo.com>
Message-ID: <1381550151.97473.YahooMailNeo@web140604.mail.bf1.yahoo.com>
Date: Fri, 11 Oct 2013 20:55:51 -0700 (PDT)
From: lars hofhansl <larsh@apache.org>
Reply-To: lars hofhansl <larsh@apache.org>
Subject: Re: skipKVsNewerThanReadpoint in StoreFileScanner and MemStoreScanner
To: "dev@hbase.apache.org" <dev@hbase.apache.org>,
  lars hofhansl <larsh@apache.org>
In-Reply-To: <1381541319.43070.YahooMailNeo@web140603.mail.bf1.yahoo.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Taking a quick look... It is easy to reduce the ThreadLocal calls by 50% wi=
thout any refactoring in the Memstore.=0A=0AAlso thinking about this more, =
we already have metadata in HFile about the newest memstoreTS; I used that =
optimize storing the memstoreTS in HFiles away (HBASE-8166 and HBASE-8151).=
=0A=0AWe could use this during scanning as well and avoid checking against =
the readpoint altogether.=0A(might need some refactoring, though, have to c=
hecked closer).=0A=0A=0A-- Lars=0A=0A=0A=0A----- Original Message -----=0AF=
rom: lars hofhansl <larsh@apache.org>=0ATo: "dev@hbase.apache.org" <dev@hba=
se.apache.org>=0ACc: =0ASent: Friday, October 11, 2013 6:28 PM=0ASubject: R=
e: skipKVsNewerThanReadpoint in StoreFileScanner and MemStoreScanner=0A=0AI=
nteresting. Never came up in my profiling of raw scan performance.=0AOf cou=
rse my assumption was that the vast majority of all KVs are in HFiles and t=
hus in the blockcache (not the memstore).=0A=0AThreadLocals are indeed expe=
nsive, we ran into such problems as well (code unrelated to HBase, though).=
=0A=0A=0AMind filing a jira, Vladimir? If there's a perf issue, we should f=
ix it.=0A=0A=0A-- Lars=0A=0A________________________________=0A=0AFrom: Vla=
dimir Rodionov <vladrodionov@gmail.com>=0ATo: "dev@hbase.apache.org" <dev@h=
base.apache.org> =0ASent: Friday, October 11, 2013 2:36 PM=0ASubject: Re: s=
kipKVsNewerThanReadpoint in StoreFileScanner and MemStoreScanner=0A=0A=0AYe=
s, threadLocal.get() is expensive when you count every nanosecond. There=0A=
is no need to call it every time in MemStoreScanner.next().=0A=0A=0A=0AOn F=
ri, Oct 11, 2013 at 2:32 PM, Ted Yu <yuzhihong@gmail.com> wrote:=0A=0A> Mul=
tiVersionConsistencyControl.getThreadReadPoint() retrieves value from a=0A>=
 ThreadLocal.=0A>=0A> Can you specify which operation becomes expensive ?=
=0A>=0A> BTW I only found IsolationLevel.READ_UNCOMMITTED=0A> in TestRowPro=
cessorEndpoint. Looks like it is not normally used.=0A>=0A>=0A> On Fri, Oct=
 11, 2013 at 2:20 PM, Vladimir Rodionov=0A> <vrodionov@carrieriq.com>wrote:=
=0A>=0A> > Update.=0A> >=0A> > This method is not called in MemStoreScanner=
, but direct call to :=0A> >=0A> > readPoint =3D MultiVersionConsistencyCon=
trol.getThreadReadPoint();=0A> >=0A> > on every next, seek etc=0A> > which =
is basically the same.=0A> >=0A> > All these calls become very expensive wh=
en you try to achieve 10s of=0A> > millions RPS in scan operations.=0A> >=
=0A> > Best regards,=0A> > Vladimir Rodionov=0A> > Principal Platform Engin=
eer=0A> > Carrier IQ, www.carrieriq.com=0A> > e-mail: vrodionov@carrieriq.c=
om=0A> >=0A> > ________________________________________=0A> > From: Vladimi=
r Rodionov=0A> > Sent: Friday, October 11, 2013 2:18 PM=0A> > To: dev@hbase=
.apache.org=0A> > Subject: skipKVsNewerThanReadpoint in StoreFileScanner an=
d=0A> MemStoreScanner=0A> >=0A> > From HRegion:=0A> >=0A> > synchronized(sc=
annerReadPoints) {=0A> >=A0 =A0 =A0 =A0=A0=A0if (isolationLevel =3D=3D Isol=
ationLevel.READ_UNCOMMITTED) {=0A> >=A0 =A0 =A0 =A0 =A0=A0=A0// This scan c=
an read even uncommitted transactions=0A> >=A0 =A0 =A0 =A0 =A0=A0=A0this.re=
adPt =3D Long.MAX_VALUE;=0A> >=A0 =A0 =A0 =A0 =A0=A0=A0MultiVersionConsiste=
ncyControl.setThreadReadPoint(this.readPt);=0A> >=A0 =A0 =A0 =A0=A0=A0} els=
e {=0A> >=A0 =A0 =A0 =A0 =A0=A0=A0this.readPt =3D=0A> > MultiVersionConsist=
encyControl.resetThreadReadPoint(mvcc);=0A> >=A0 =A0 =A0 =A0=A0=A0}=0A> >=
=0A> > If isolation level is READ_UNCOMMITTED than this method should never=
 be=0A> > called in both StoreFileScanner and MemStoreScanner=0A> > If isol=
ation level is READ_COMMITTED than this method must be called only=0A> > in=
 MemStoreScanner, because HFiles are immutable no new KV's can be added=0A>=
 > to HFile after StoreFileScanner was created.=0A> >=0A> >=0A> > Correct?=
=0A> >=0A> >=0A> > Best regards,=0A> > Vladimir Rodionov=0A> > Principal Pl=
atform Engineer=0A> > Carrier IQ, www.carrieriq.com=0A> > e-mail: vrodionov=
@carrieriq.com=0A> >=0A> > Confidentiality Notice:=A0 The information conta=
ined in this message,=0A> > including any attachments hereto, may be confid=
ential and is intended to=0A> be=0A> > read only by the individual or entit=
y to whom this message is addressed.=0A> If=0A> > the reader of this messag=
e is not the intended recipient or an agent or=0A> > designee of the intend=
ed recipient, please note that any review, use,=0A> > disclosure or distrib=
ution of this message or its attachments, in any=0A> form,=0A> > is strictl=
y prohibited.=A0 If you have received this message in error,=0A> please=0A>=
 > immediately notify the sender and/or Notifications@carrieriq.com and=0A>=
 > delete or destroy any copy of this message and its attachments.=0A> >=0A=
>