Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
References: 
 <CAOT3TWptjL+XHVXzN9J+PQU-H=r_nM485Ei83oWHdr3Lymt80Q@mail.gmail.com>
 <1310062928.68801.YahooMailNeo@web65515.mail.ac4.yahoo.com>
 <CAOT3TWqXgLiCtoVfPxWeV4zA2dPXth4ibtC8SvUgEp4bp5tTBw@mail.gmail.com>
Message-ID: <1310065878.43031.YahooMailNeo@web65507.mail.ac4.yahoo.com>
Date: Thu, 7 Jul 2011 12:11:18 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: Andrew Purtell <apurtell@apache.org>
Subject: Re: Hbase performance with HDFS
To: Mohit Anchlia <mohitanchlia@gmail.com>,
  "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: 
 <CAOT3TWqXgLiCtoVfPxWeV4zA2dPXth4ibtC8SvUgEp4bp5tTBw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1881162354-1310065878=:43031"

--0-1881162354-1310065878=:43031
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Some thoughts off the top of my head. Lars' architecture material might/sho=
uld cover this too. Pretty=A0sure his book will.=A0=0A=0ARegarding reads:=
=0A=0AOne does not have to read a whole HDFS block. You can request arbitra=
ry byte ranges with the block, via positioned reads. (It is true also that =
HDFS can be improved for better random reading performance in ways not nece=
ssarily yet committed to trunk or especially a 0.20.x branch with append su=
pport for HBase. See=A0https://issues.apache.org/jira/browse/HDFS-1323)=0A=
=0AHBase holds indexes to store files in HDFS in memory. We also open all s=
tore files at the HDFS layer and stash those references. Additionally, user=
s can specify the use of bloom filters to improve query time performance th=
rough wholesale skipping of HFile reads if they are known not to contain da=
ta that satisfies the query. Bloom filters are held in memory as well.=0A=
=0ASo with indexes resident in memory when handling Gets we know the byte r=
anges within HDFS block(s) that contain the data of interest. With position=
ed reads we retrieve only those bytes from a DataNode. With optional bloomf=
ilters we avoid whole HFiles entirely.=0A=0ARegarding writes:=0A=0AI think =
you should consult the bigtable paper again if you are still asking about t=
he write path. The database is log structured. Writes are accumulated in me=
mory, and flushed all at once. Later flush files are compacted as needed, b=
ecause as you point out GFS and HDFS are optimized for streaming sequential=
 reads and writes.=0A=0A=0ABest regards,=0A=0A=0A=A0 - Andy=0A=0AProblems w=
orthy of attack prove their worth by hitting back. - Piet Hein (via Tom Whi=
te)=0A=0A=0A>________________________________=0A>From: Mohit Anchlia <mohit=
anchlia@gmail.com>=0A>To: user@hbase.apache.org; Andrew Purtell <apurtell@a=
pache.org>=0A>Sent: Thursday, July 7, 2011 11:53 AM=0A>Subject: Re: Hbase p=
erformance with HDFS=0A>=0A>I have looked at bigtable and it's ssTables etc=
. But my question is=0A>directly related to how it's used with HDFS. HDFS r=
ecommends large=0A>files, bigger blocks, write once and read many sequentia=
l reads. But=0A>accessing small rows and writing small rows is more random =
and=0A>different than inherent design of HDFS. How do these 2 go together a=
nd=0A>is able to provide performance.=0A>=0A>On Thu, Jul 7, 2011 at 11:22 A=
M, Andrew Purtell <apurtell@apache.org> wrote:=0A>> Hi Mohit,=0A>>=0A>> Sta=
rt here:=A0http://labs.google.com/papers/bigtable.html=0A>>=0A>> Best regar=
ds,=0A>>=0A>>=0A>> =A0 =A0 - Andy=0A>>=0A>> Problems worthy of attack prove=
 their worth by hitting back. - Piet Hein (via Tom White)=0A>>=0A>>=0A>>>__=
______________________________=0A>>>From: Mohit Anchlia <mohitanchlia@gmail=
.com>=0A>>>To: user@hbase.apache.org=0A>>>Sent: Thursday, July 7, 2011 11:1=
2 AM=0A>>>Subject: Hbase performance with HDFS=0A>>>=0A>>>I've been trying =
to understand how Hbase can provide good performance=0A>>>using HDFS when p=
urpose of HDFS is sequential large block sizes which=0A>>>is inherently dif=
ferent than of Hbase where it's more random and row=0A>>>sizes might be ver=
y small.=0A>>>=0A>>>I am reading this but doesn't answer my question. It do=
es say that=0A>>>HFile block size is different but how it really works with=
 HDFS is=0A>>>what I am trying to understand.=0A>>>=0A>>>http://www.larsgeo=
rge.com/2009/10/hbase-architecture-101-storage.html=0A>>>=0A>>>=0A>>>=0A>=
=0A>=0A>
--0-1881162354-1310065878=:43031--