Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 209.85.221.184 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:date
         :x-google-sender-auth:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=Wire7/aAUnnUbobXnrSwpA4a6jEyKm94pXWlCNS0xB0mNh4QJpY6npaQlhpVn1CHPA
         DxQ+B3TBMfQSLUndzVgPsP8KbZGTNh8VbhrLviQxZZIs+JSXnryShvTEIzQcGqUMVf6s
         HVvNtUcpRc7VxXvWqkR1YHeZblWawZ0ssa7rk=
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <1266346222.15001.455.camel@puma>
References: <1266277557.15001.241.camel@puma>
 <1266300300.15001.306.camel@puma>
	 <7c962aed1002152214l60d67236mc4daa94e11631095@mail.gmail.com>
	 <1266302566.15001.327.camel@puma>
	 <7c457ebe1002152248o5fef7131p1d5599c8d0dd3e34@mail.gmail.com>
	 <1266303245.15001.332.camel@puma>
	 <7c457ebe1002152311i776f871cveedf093006fe287f@mail.gmail.com>
	 <1266304866.15001.348.camel@puma>
	 <7c962aed1002161017p4b64349ekec554f8505763ee0@mail.gmail.com>
	 <1266346222.15001.455.camel@puma>
Date: Tue, 16 Feb 2010 12:05:55 -0800
Message-ID: <7c962aed1002161205n4cbb6833n478c8121a1f6d396@mail.gmail.com>
Subject: Re: Optimizations for random read performance
From: Stack <stack@duboce.net>
To: hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 16, 2010 at 10:50 AM, James Baldassari <james@dataxu.com> wrote=
:
> Today we added a fourth region server and forced the data to be
> redistributed evenly by exporting /hbase and then importing it back in
> (the Hadoop redistribution tool didn't even out the data completely).

Did you have to do the export?   Adding the regionserver should be
enough.  Over time, it would balance out (If you wanted the migration
of data, run a major compaction).

> We also increased the max heap size on the region server from 4G to 5G
> and decreased the block cache value from 0.4 back to 0.2. =A0These change=
s
> didn't seem to provide any real benefit, unfortunately. =A0In a few
> minutes we're going to try deploying the patched (HBASE-2180) HBase
> 0.20.3. =A0If all of these changes don't work we'll look at tweaking the
> block size as you suggested.
>
> LZO is not an option for us due to the GPL license on the required Java
> connector library (http://code.google.com/p/hadoop-gpl-compression/).
> Another here group was looking at using LZO in their Hadoop cluster but
> ruled it out due to the licensing issues. =A0Is anyone aware of any LZO
> (or other codec) libraries/connectors released under the Apache or LGPL
> licenses that can be used with HBase/Hadoop?

Should only be a matter if you intend distributing the above.

>
> We do have Ganglia with monitoring and alerts. =A0We're not swapping righ=
t
> now, although it is possible for that to happen at some point. =A0What
> seems to be happening is that the load on all region servers will be
> fairly low (< 1.0) except for one region server which will have a load
> of around 10 with a high I/O wait. =A0Then this region server will go bac=
k
> to normal and a different region server will experience high load and
> I/O wait. =A0I'm hopeful that HBASE-2180 may resolve some of these issues=
.
> I'll send an update after we deploy the patched jar.
>

What is it about your reads that they are all coming into the one
regionserver only?  Whats your key schema look like?  Are they all
hitting the one region?

Thanks James,
St.Ack


> -James
>
>
> On Tue, 2010-02-16 at 12:17 -0600, Stack wrote:
>> If you don't do lzo, and if your cell size is smallish, then try a
>> different block size. =A0Default blocksize is 64k which might be ok for
>> a single-seek -- i.e. costs near same getting 64k as it does 4k -- but
>> for a random-read loading with lots of concurrency, it might make for
>> more work being done than needs be and so throughput drops. =A0To enable
>> 4k blocks, do as Lars said for enabling lzo only change the block size
>> when table is offline. =A0 You could run a major compaction and it'll
>> rewrite all as 4k blocks promptly (at a large i/o cost) or just let
>> the cluster go about its business and as it compacts naturally, the
>> new files will be 4k.
>>
>> In an earlier note you say you are over-allocated and so you could be
>> swapping (Are you? =A0Do you ops teams have ganglia or some such running
>> against this cluster?). =A0 A JVM whose heap is paging will not perform
>> at all. =A0You don't even hit the max on all daemons for paging to
>> happen. =A0See "Are you swapping" in this page
>> http://wiki.apache.org/hadoop/PerformanceTuning.
>>
>> St.Ack
>>
>>
>> On Mon, Feb 15, 2010 at 11:21 PM, James Baldassari <james@dataxu.com> wr=
ote:
>> > No, we don't have LZO on the table right now. =A0I guess that's someth=
ing
>> > else that we can try. =A0I'll ask our ops team if we can steal another
>> > node or two for the cluster if you think that will help. =A0I'll repor=
t
>> > back with results as soon as I can. =A0Thanks again for working with m=
e on
>> > this! =A0This is definitely the most responsive users list I've ever
>> > posted to.
>> >
>> > -James
>> >
>> >
>> > On Tue, 2010-02-16 at 01:11 -0600, Dan Washusen wrote:
>> >> You could just add another node to your cluster to solve the immediat=
e
>> >> problem. =A0Then keep an eye on load, etc to preemptively add more no=
des as
>> >> needed?
>> >>
>> >> Out of interest do you have LZO compression enabled on your table? =
=A0That
>> >> makes the block cache and IO ops much more effective...
>> >>
>> >> Regarding GC logging:
>> >> Also, another option for GC logging is 'jstat'. =A0For example, runni=
ng the
>> >> following command will print out the VM heap utilization every 1 seco=
nd:
>> >>
>> >> > jstat -gcutil <pid> 1000
>> >> >
>> >>
>> >> The last column shows total amount of time (in seconds) spent garbage
>> >> collecting. =A0 You want to see very small increments... =A0The other
>> >> interesting columns are "O" and "E". =A0They show the percentage of O=
ld and
>> >> Eden used. =A0If old gen is staying up in the high 90's then there ar=
e more
>> >> long lived objects then available memory...
>> >>
>> >> Cheers,
>> >> Dan
>> >>
>> >> On 16 February 2010 17:54, James Baldassari <james@dataxu.com> wrote:
>> >>
>> >> > How much should I give the region servers? =A0That machine is alrea=
dy
>> >> > overallocated, by which I mean that the sum of the max heap sizes o=
f all
>> >> > java processes running there is greater than the amount physical me=
mory,
>> >> > which can lead to swapping. =A0We have: Hadoop data node, Hadoop ta=
sk
>> >> > tracker, ZooKeeper peer, and region server. =A0The machine has 8G o=
f
>> >> > physical memory. =A0The region server currently has a max heap size=
 of 4G.
>> >> > Should I increase to 6G? =A0Should I decrease the block cache back =
down to
>> >> > 20% or even lower? =A0Do we need to move to a 16G server?
>> >> >
>> >> > Thanks,
>> >> > James
>> >> >
>> >> >
>> >> > On Tue, 2010-02-16 at 00:48 -0600, Dan Washusen wrote:
>> >> > > 32% IO on region server 3! =A0Ouch! :)
>> >> > >
>> >> > > Increasing the block cache to 40% of VM memory without upping the=
 total
>> >> > > available memory may only exacerbated the issue. =A0I notice that=
 region
>> >> > > server 2 was already using 3300mb of the 4000mb heap. By increasi=
ng the
>> >> > > block cache size to 40% you have now given the block cache 1600mb
>> >> > compared
>> >> > > to the previous 800mb...
>> >> > >
>> >> > > Can you give the region servers more memory?
>> >> > >
>> >> > > Cheers,
>> >> > > Dan
>> >> > >
>> >> > > On 16 February 2010 17:42, James Baldassari <james@dataxu.com> wr=
ote:
>> >> > >
>> >> > > > On Tue, 2010-02-16 at 00:14 -0600, Stack wrote:
>> >> > > > > On Mon, Feb 15, 2010 at 10:05 PM, James Baldassari <james@dat=
axu.com
>> >> > >
>> >> > > > wrote:
>> >> > > > > > =A0Applying HBASE-2180 isn't really an option at this
>> >> > > > > > time because we've been told to stick with the Cloudera dis=
tro.
>> >> > > > >
>> >> > > > > I'm sure the wouldn't mind (smile). =A0Seems to about double
>> >> > throughput.
>> >> > > >
>> >> > > > Hmm, well I might be able to convince them ;)
>> >> > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > > If I had to guess, I would say the performance issues start=
 to
>> >> > happen
>> >> > > > > > around the time the region servers hit max heap size, which=
 occurs
>> >> > > > > > within minutes of exposing the app to live traffic. =A0Coul=
d GC be
>> >> > > > killing
>> >> > > > > > us? =A0We use the concurrent collector as suggested. =A0I s=
aw on the
>> >> > > > > > performance page some mention of limiting the size of the n=
ew
>> >> > > > generation
>> >> > > > > > like -XX:NewSize=3D6m -XX:MaxNewSize=3D6m. =A0Is that worth=
 trying?
>> >> > > > >
>> >> > > > > Enable GC logging for a while? =A0See hbase-env.sh. =A0Uncomm=
ent this
>> >> > line:
>> >> > > > >
>> >> > > > > # export HBASE_OPTS=3D"$HBASE_OPTS -verbose:gc -XX:+PrintGCDe=
tails
>> >> > > > > XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
>> >> > > >
>> >> > > > I did uncomment that line, but I can't figure out where the
>> >> > gc-hbase.log
>> >> > > > is. =A0It's not with the other logs. =A0When starting HBase the=
 GC output
>> >> > > > seems to be going to stdout rather than the file. =A0Maybe a Cl=
oudera
>> >> > > > thing. =A0I'll do some digging.
>> >> > > >
>> >> > > > >
>> >> > > > > You are using recent JVM? =A01.6.0_10 or greater? =A01.6.0_18=
 might have
>> >> > > > issues.
>> >> > > >
>> >> > > > We're on 1.6.0_16 at the moment.
>> >> > > >
>> >> > > > >
>> >> > > > > Whats CPU and iowait or wa in top look like on these machines=
,
>> >> > > > > particularly the loaded machine?
>> >> > > > >
>> >> > > > > How many disks in the machines?
>> >> > > >
>> >> > > > I'll have to ask our ops guys about the disks. =A0The high load=
 has now
>> >> > > > switched from region server 1 to 3. =A0I just saw in our logs t=
hat it
>> >> > took
>> >> > > > 139383.065 milliseconds to do 5000 gets, ~36 gets/second, ouch.=
 =A0Here
>> >> > > > are the highlights from top for each region server:
>> >> > > >
>> >> > > > Region Server 1:
>> >> > > > top - 01:39:41 up 4 days, 13:44, =A04 users, =A0load average: 1=
.89, 0.99,
>> >> > 1.19
>> >> > > > Tasks: 194 total, =A0 1 running, 193 sleeping, =A0 0 stopped, =
=A0 0 zombie
>> >> > > > Cpu(s): 15.6%us, =A05.8%sy, =A00.0%ni, 76.9%id, =A00.0%wa, =A00=
.1%hi, =A01.6%si,
>> >> > > > =A00.0%st
>> >> > > > Mem: =A0 8166588k total, =A08112812k used, =A0 =A053776k free, =
=A0 =A0 8832k
>> >> > buffers
>> >> > > > Swap: =A01052248k total, =A0 =A0 =A0152k used, =A01052096k free=
, =A02831076k cached
>> >> > > > =A0PID USER =A0 =A0 =A0PR =A0NI =A0VIRT =A0RES =A0SHR S %CPU %M=
EM =A0 =A0TIME+ =A0COMMAND
>> >> > > > 21961 hadoop =A0 =A019 =A0 0 4830m 4.2g =A010m S 114.3 53.6 =A0=
37:26.58 java
>> >> > > > 21618 hadoop =A0 =A021 =A0 0 4643m 578m 9804 S 66.1 =A07.3 =A01=
9:06.89 java
>> >> > > >
>> >> > > > Region Server 2:
>> >> > > > top - 01:40:28 up 4 days, 13:43, =A04 users, =A0load average: 3=
.93, 2.17,
>> >> > 1.39
>> >> > > > Tasks: 194 total, =A0 1 running, 193 sleeping, =A0 0 stopped, =
=A0 0 zombie
>> >> > > > Cpu(s): 11.3%us, =A03.1%sy, =A00.0%ni, 83.4%id, =A01.2%wa, =A00=
.1%hi, =A00.9%si,
>> >> > > > =A00.0%st
>> >> > > > Mem: =A0 8166588k total, =A07971572k used, =A0 195016k free, =
=A0 =A034972k
>> >> > buffers
>> >> > > > Swap: =A01052248k total, =A0 =A0 =A0152k used, =A01052096k free=
, =A02944712k cached
>> >> > > > =A0PID USER =A0 =A0 =A0PR =A0NI =A0VIRT =A0RES =A0SHR S %CPU %M=
EM =A0 =A0TIME+ =A0COMMAND
>> >> > > > 15752 hadoop =A0 =A018 =A0 0 4742m 4.1g =A010m S 210.6 53.1 =A0=
41:52.80 java
>> >> > > > 15405 hadoop =A0 =A020 =A0 0 4660m 317m 9800 S 114.0 =A04.0 =A0=
27:34.17 java
>> >> > > >
>> >> > > > Region Server 3:
>> >> > > > top - 01:40:35 up 2 days, =A09:04, =A04 users, =A0load average:=
 10.15, 11.05,
>> >> > > > 11.79
>> >> > > > Tasks: 195 total, =A0 1 running, 194 sleeping, =A0 0 stopped, =
=A0 0 zombie
>> >> > > > Cpu(s): 28.7%us, 10.1%sy, =A00.0%ni, 25.8%id, 32.9%wa, =A00.1%h=
i, =A02.4%si,
>> >> > > > =A00.0%st
>> >> > > > Mem: =A0 8166572k total, =A08118592k used, =A0 =A047980k free, =
=A0 =A0 3264k
>> >> > buffers
>> >> > > > Swap: =A01052248k total, =A0 =A0 =A0140k used, =A01052108k free=
, =A02099896k cached
>> >> > > > =A0PID USER =A0 =A0 =A0PR =A0NI =A0VIRT =A0RES =A0SHR S %CPU %M=
EM =A0 =A0TIME+ =A0COMMAND
>> >> > > > 15636 hadoop =A0 =A018 =A0 0 4806m 4.2g =A010m S 206.9 53.3 =A0=
87:48.81 java
>> >> > > > 15243 hadoop =A0 =A018 =A0 0 4734m 1.3g 9800 S 117.6 16.7 =A063=
:46.52 java
>> >> > > >
>> >> > > > -James
>> >> > > >
>> >> > > > >
>> >> > > > > St>Ack
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > >
>> >> > > > > > Here are the new region server stats along with load averag=
es:
>> >> > > > > >
>> >> > > > > > Region Server 1:
>> >> > > > > > request=3D0.0, regions=3D16, stores=3D16, storefiles=3D33,
>> >> > > > storefileIndexSize=3D4, memstoreSize=3D1, compactionQueueSize=
=3D0,
>> >> > usedHeap=3D2891,
>> >> > > > maxHeap=3D4079, blockCacheSize=3D1403878072, blockCacheFree=3D3=
07135816,
>> >> > > > blockCacheCount=3D21107, blockCacheHitRatio=3D84, fsReadLatency=
=3D0,
>> >> > > > fsWriteLatency=3D0, fsSyncLatency=3D0
>> >> > > > > > Load Averages: 10.34, 10.58, 7.08
>> >> > > > > >
>> >> > > > > > Region Server 2:
>> >> > > > > > request=3D0.0, regions=3D15, stores=3D16, storefiles=3D26,
>> >> > > > storefileIndexSize=3D3, memstoreSize=3D1, compactionQueueSize=
=3D0,
>> >> > usedHeap=3D3257,
>> >> > > > maxHeap=3D4079, blockCacheSize=3D661765368, blockCacheFree=3D19=
3741576,
>> >> > > > blockCacheCount=3D9942, blockCacheHitRatio=3D77, fsReadLatency=
=3D0,
>> >> > > > fsWriteLatency=3D0, fsSyncLatency=3D0
>> >> > > > > > Load Averages: 1.90, 1.23, 0.98
>> >> > > > > >
>> >> > > > > > Region Server 3:
>> >> > > > > > request=3D0.0, regions=3D16, stores=3D16, storefiles=3D41,
>> >> > > > storefileIndexSize=3D4, memstoreSize=3D4, compactionQueueSize=
=3D0,
>> >> > usedHeap=3D1627,
>> >> > > > maxHeap=3D4079, blockCacheSize=3D665117184, blockCacheFree=3D19=
0389760,
>> >> > > > blockCacheCount=3D9995, blockCacheHitRatio=3D70, fsReadLatency=
=3D0,
>> >> > > > fsWriteLatency=3D0, fsSyncLatency=3D0
>> >> > > > > > Load Averages: 2.01, 3.56, 4.18
>> >> > > > > >
>> >> > > > > > That first region server is getting hit much harder than th=
e
>> >> > others.
>> >> > > > > > They're identical machines (8-core), and the distribution o=
f keys
>> >> > > > should
>> >> > > > > > be fairly random, so I'm not sure why that would happen. =
=A0Any other
>> >> > > > > > ideas or suggestions would be greatly appreciated.
>> >> > > > > >
>> >> > > > > > Thanks,
>> >> > > > > > James
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Mon, 2010-02-15 at 21:51 -0600, Stack wrote:
>> >> > > > > >> Yeah, I was going to say that if your loading is mostly re=
ad, you
>> >> > can
>> >> > > > > >> probably go up from the 0.2 given over to cache. =A0I like=
 Dan's
>> >> > > > > >> suggestion of trying it first on one server, if you can.
>> >> > > > > >>
>> >> > > > > >> St.Ack
>> >> > > > > >>
>> >> > > > > >> On Mon, Feb 15, 2010 at 5:22 PM, Dan Washusen <dan@reactiv=
e.org>
>> >> > > > wrote:
>> >> > > > > >> > So roughly 72% of reads use the blocks held in the block
>> >> > cache...
>> >> > > > > >> >
>> >> > > > > >> > It would be interesting to see the difference between wh=
en it
>> >> > was
>> >> > > > working OK
>> >> > > > > >> > and now. =A0Could you try increasing the memory allocate=
d to one
>> >> > of
>> >> > > > the
>> >> > > > > >> > regions and also increasing the "hfile.block.cache.size"=
 to say
>> >> > > > '0.4' on the
>> >> > > > > >> > same region?
>> >> > > > > >> >
>> >> > > > > >> > On 16 February 2010 11:54, James Baldassari <james@datax=
u.com>
>> >> > > > wrote:
>> >> > > > > >> >
>> >> > > > > >> >> Hi Dan. =A0Thanks for your suggestions. =A0I am doing w=
rites at the
>> >> > > > same
>> >> > > > > >> >> time as reads, but there are usually many more reads th=
an
>> >> > writes.
>> >> > > > =A0Here
>> >> > > > > >> >> are the stats for all three region servers:
>> >> > > > > >> >>
>> >> > > > > >> >> Region Server 1:
>> >> > > > > >> >> request=3D0.0, regions=3D15, stores=3D16, storefiles=3D=
34,
>> >> > > > storefileIndexSize=3D3,
>> >> > > > > >> >> memstoreSize=3D308, compactionQueueSize=3D0, usedHeap=
=3D3096,
>> >> > > > maxHeap=3D4079,
>> >> > > > > >> >> blockCacheSize=3D705474544, blockCacheFree=3D150032400,
>> >> > > > blockCacheCount=3D10606,
>> >> > > > > >> >> blockCacheHitRatio=3D76, fsReadLatency=3D0, fsWriteLate=
ncy=3D0,
>> >> > > > fsSyncLatency=3D0
>> >> > > > > >> >>
>> >> > > > > >> >> Region Server 2:
>> >> > > > > >> >> request=3D0.0, regions=3D16, stores=3D16, storefiles=3D=
39,
>> >> > > > storefileIndexSize=3D4,
>> >> > > > > >> >> memstoreSize=3D225, compactionQueueSize=3D0, usedHeap=
=3D3380,
>> >> > > > maxHeap=3D4079,
>> >> > > > > >> >> blockCacheSize=3D643172800, blockCacheFree=3D212334144,
>> >> > > > blockCacheCount=3D9660,
>> >> > > > > >> >> blockCacheHitRatio=3D69, fsReadLatency=3D0, fsWriteLate=
ncy=3D0,
>> >> > > > fsSyncLatency=3D0
>> >> > > > > >> >>
>> >> > > > > >> >> Region Server 3:
>> >> > > > > >> >> request=3D0.0, regions=3D13, stores=3D13, storefiles=3D=
31,
>> >> > > > storefileIndexSize=3D4,
>> >> > > > > >> >> memstoreSize=3D177, compactionQueueSize=3D0, usedHeap=
=3D1905,
>> >> > > > maxHeap=3D4079,
>> >> > > > > >> >> blockCacheSize=3D682848608, blockCacheFree=3D172658336,
>> >> > > > blockCacheCount=3D10262,
>> >> > > > > >> >> blockCacheHitRatio=3D72, fsReadLatency=3D0, fsWriteLate=
ncy=3D0,
>> >> > > > fsSyncLatency=3D0
>> >> > > > > >> >>
>> >> > > > > >> >> The average blockCacheHitRatio is about 72. =A0Is this =
too low?
>> >> > > > =A0Anything
>> >> > > > > >> >> else I can check?
>> >> > > > > >> >>
>> >> > > > > >> >> -James
>> >> > > > > >> >>
>> >> > > > > >> >>
>> >> > > > > >> >> On Mon, 2010-02-15 at 18:16 -0600, Dan Washusen wrote:
>> >> > > > > >> >> > Maybe the block cache is thrashing?
>> >> > > > > >> >> >
>> >> > > > > >> >> > If you are regularly writing data to your tables then=
 it's
>> >> > > > possible that
>> >> > > > > >> >> the
>> >> > > > > >> >> > block cache is no longer being effective. =A0On the r=
egion
>> >> > server
>> >> > > > web UI
>> >> > > > > >> >> check
>> >> > > > > >> >> > the blockCacheHitRatio value. =A0You want this value =
to be high
>> >> > (0
>> >> > > > - 100).
>> >> > > > > >> >> =A0If
>> >> > > > > >> >> > this value is low it means that HBase has to go to di=
sk to
>> >> > fetch
>> >> > > > blocks
>> >> > > > > >> >> of
>> >> > > > > >> >> > data. =A0You can control the amount of VM memory that=
 HBase
>> >> > > > allocates to
>> >> > > > > >> >> the
>> >> > > > > >> >> > block cache using the "hfile.block.cache.size" proper=
ty
>> >> > (default
>> >> > > > is 0.2
>> >> > > > > >> >> > (20%)).
>> >> > > > > >> >> >
>> >> > > > > >> >> > Cheers,
>> >> > > > > >> >> > Dan
>> >> > > > > >> >> >
>> >> > > > > >> >> > On 16 February 2010 10:45, James Baldassari <
>> >> > james@dataxu.com>
>> >> > > > wrote:
>> >> > > > > >> >> >
>> >> > > > > >> >> > > Hi,
>> >> > > > > >> >> > >
>> >> > > > > >> >> > > Does anyone have any tips to share regarding optimi=
zation
>> >> > for
>> >> > > > random
>> >> > > > > >> >> > > read performance? =A0For writes I've found that set=
ting a
>> >> > large
>> >> > > > write
>> >> > > > > >> >> > > buffer and setting auto-flush to false on the clien=
t side
>> >> > > > significantly
>> >> > > > > >> >> > > improved put performance. =A0Are there any similar =
easy
>> >> > tweaks to
>> >> > > > improve
>> >> > > > > >> >> > > random read performance?
>> >> > > > > >> >> > >
>> >> > > > > >> >> > > I'm using HBase 0.20.3 in a very read-heavy real-ti=
me
>> >> > system
>> >> > > > with 1
>> >> > > > > >> >> > > master and 3 region servers. =A0It was working ok f=
or a
>> >> > while,
>> >> > > > but today
>> >> > > > > >> >> > > there was a severe degradation in read performance.
>> >> > =A0Restarting
>> >> > > > Hadoop
>> >> > > > > >> >> > > and HBase didn't help, are there are no errors in t=
he logs.
>> >> > > > =A0Read
>> >> > > > > >> >> > > performance starts off around 1,000-2,000 gets/seco=
nd but
>> >> > > > quickly
>> >> > > > > >> >> > > (within minutes) drops to around 100 gets/second.
>> >> > > > > >> >> > >
>> >> > > > > >> >> > > I've already looked at the performance tuning wiki =
page.
>> >> > =A0On
>> >> > > > the server
>> >> > > > > >> >> > > side I've increased hbase.regionserver.handler.coun=
t from
>> >> > 10 to
>> >> > > > 100,
>> >> > > > > >> >> but
>> >> > > > > >> >> > > it didn't help. =A0Maybe this is expected because I=
'm only
>> >> > using
>> >> > > > a single
>> >> > > > > >> >> > > client to do reads. =A0I'm working on implementing =
a client
>> >> > pool
>> >> > > > now, but
>> >> > > > > >> >> > > I'm wondering if there are any other settings on th=
e server
>> >> > or
>> >> > > > client
>> >> > > > > >> >> > > side that might improve things.
>> >> > > > > >> >> > >
>> >> > > > > >> >> > > Thanks,
>> >> > > > > >> >> > > James
>> >> > > > > >> >> > >
>> >> > > > > >> >> > >
>> >> > > > > >> >> > >
>> >> > > > > >> >>
>> >> > > > > >> >>
>> >> > > > > >> >
>> >> > > > > >
>> >> > > > > >
>> >> > > >
>> >> > > >
>> >> >
>> >> >
>> >
>> >
>
>