hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Baldassari <ja...@dataxu.com>
Subject Re: Optimizations for random read performance
Date Tue, 16 Feb 2010 22:25:31 GMT
On Tue, 2010-02-16 at 14:05 -0600, Stack wrote:
> On Tue, Feb 16, 2010 at 10:50 AM, James Baldassari <james@dataxu.com> wrote:
> > Today we added a fourth region server and forced the data to be
> > redistributed evenly by exporting /hbase and then importing it back in
> > (the Hadoop redistribution tool didn't even out the data completely).
> 
> Did you have to do the export?   Adding the regionserver should be
> enough.  Over time, it would balance out (If you wanted the migration
> of data, run a major compaction).

Yes, over time it would balance out, but we were trying to solve the
immediate problem of uneven load across the region servers, so we wanted
to make sure that all the data was evenly distributed to eliminate that
as a potential cause.  We did do the export and import, and then the
data was evenly distributed, so that's not the issue.

Whether the keys themselves are evenly distributed is another matter.
Our keys are user IDs, and they should be fairly random.  If we do a
status 'detailed' in the hbase shell we see the following distribution
for the value of "requests" (not entirely sure what this value means):
hdfs01: 7078
hdfs02: 5898
hdfs03: 5870
hdfs04: 3807

There are no order of magnitude differences here, and the request count
doesn't seem to map to the load on the server.  Right now hdfs02 has a
load of 16 while the 3 others have loads between 1 and 2.  Applying
HBASE-2180 did not make any measurable difference.  There are no errors
in the region server logs.  However, looking at the Hadoop datanode
logs, I'm seeing lots of these:

2010-02-16 17:07:54,064 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.24.183.165:50010,
storageID=DS-1519453437-10.24.183.165-50010-1265907617548, infoPort=50075, ipcPort=50020):DataXceiver
java.io.EOFException
        at java.io.DataInputStream.readShort(DataInputStream.java:298)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:79)
        at java.lang.Thread.run(Thread.java:619)

One thing I should mention is that I'm not sure exactly how many
requests/second HBase is servicing now that I've switched to using HBase
client pools.  HBase may be servicing gets at a relatively high rate,
but it's just not keeping up with the incoming requests from our
clients.  I think there may be some additional client-side optimizations
we can make, so I might take some time to look at our client code to see
if there are any problems there.  However, I do think it's strange that
the load is so unbalanced on the region servers. 

We're also going to try throwing some more hardware at the problem.
We'll set up a new cluster with 16-core, 16G nodes to see if they are
better able to handle the large number of client requests.  We might
also decrease the block size to 32k or lower.

> 
> > We also increased the max heap size on the region server from 4G to 5G
> > and decreased the block cache value from 0.4 back to 0.2.  These changes
> > didn't seem to provide any real benefit, unfortunately.  In a few
> > minutes we're going to try deploying the patched (HBASE-2180) HBase
> > 0.20.3.  If all of these changes don't work we'll look at tweaking the
> > block size as you suggested.
> >
> > LZO is not an option for us due to the GPL license on the required Java
> > connector library (http://code.google.com/p/hadoop-gpl-compression/).
> > Another here group was looking at using LZO in their Hadoop cluster but
> > ruled it out due to the licensing issues.  Is anyone aware of any LZO
> > (or other codec) libraries/connectors released under the Apache or LGPL
> > licenses that can be used with HBase/Hadoop?
> 
> Should only be a matter if you intend distributing the above.

This is probably a topic for a separate thread, but I've never seen a
legal definition for the word "distribution."  How does this apply to
the SaaS model?

> 
> >
> > We do have Ganglia with monitoring and alerts.  We're not swapping right
> > now, although it is possible for that to happen at some point.  What
> > seems to be happening is that the load on all region servers will be
> > fairly low (< 1.0) except for one region server which will have a load
> > of around 10 with a high I/O wait.  Then this region server will go back
> > to normal and a different region server will experience high load and
> > I/O wait.  I'm hopeful that HBASE-2180 may resolve some of these issues.
> > I'll send an update after we deploy the patched jar.
> >
> 
> What is it about your reads that they are all coming into the one
> regionserver only?  Whats your key schema look like?  Are they all
> hitting the one region?

I wish I knew the answer to this.  As I mentioned above, our keys are
user IDs.  They come in at random times, not in chunks of similar IDs.

-James


> 
> Thanks James,
> St.Ack
> 
> 
> > -James
> >
> >
> > On Tue, 2010-02-16 at 12:17 -0600, Stack wrote:
> >> If you don't do lzo, and if your cell size is smallish, then try a
> >> different block size.  Default blocksize is 64k which might be ok for
> >> a single-seek -- i.e. costs near same getting 64k as it does 4k -- but
> >> for a random-read loading with lots of concurrency, it might make for
> >> more work being done than needs be and so throughput drops.  To enable
> >> 4k blocks, do as Lars said for enabling lzo only change the block size
> >> when table is offline.   You could run a major compaction and it'll
> >> rewrite all as 4k blocks promptly (at a large i/o cost) or just let
> >> the cluster go about its business and as it compacts naturally, the
> >> new files will be 4k.
> >>
> >> In an earlier note you say you are over-allocated and so you could be
> >> swapping (Are you?  Do you ops teams have ganglia or some such running
> >> against this cluster?).   A JVM whose heap is paging will not perform
> >> at all.  You don't even hit the max on all daemons for paging to
> >> happen.  See "Are you swapping" in this page
> >> http://wiki.apache.org/hadoop/PerformanceTuning.
> >>
> >> St.Ack
> >>
> >>
> >> On Mon, Feb 15, 2010 at 11:21 PM, James Baldassari <james@dataxu.com>
wrote:
> >> > No, we don't have LZO on the table right now.  I guess that's something
> >> > else that we can try.  I'll ask our ops team if we can steal another
> >> > node or two for the cluster if you think that will help.  I'll report
> >> > back with results as soon as I can.  Thanks again for working with me on
> >> > this!  This is definitely the most responsive users list I've ever
> >> > posted to.
> >> >
> >> > -James
> >> >
> >> >
> >> > On Tue, 2010-02-16 at 01:11 -0600, Dan Washusen wrote:
> >> >> You could just add another node to your cluster to solve the immediate
> >> >> problem.  Then keep an eye on load, etc to preemptively add more nodes
as
> >> >> needed?
> >> >>
> >> >> Out of interest do you have LZO compression enabled on your table?
 That
> >> >> makes the block cache and IO ops much more effective...
> >> >>
> >> >> Regarding GC logging:
> >> >> Also, another option for GC logging is 'jstat'.  For example, running
the
> >> >> following command will print out the VM heap utilization every 1 second:
> >> >>
> >> >> > jstat -gcutil <pid> 1000
> >> >> >
> >> >>
> >> >> The last column shows total amount of time (in seconds) spent garbage
> >> >> collecting.   You want to see very small increments...  The other
> >> >> interesting columns are "O" and "E".  They show the percentage of Old
and
> >> >> Eden used.  If old gen is staying up in the high 90's then there are
more
> >> >> long lived objects then available memory...
> >> >>
> >> >> Cheers,
> >> >> Dan
> >> >>
> >> >> On 16 February 2010 17:54, James Baldassari <james@dataxu.com>
wrote:
> >> >>
> >> >> > How much should I give the region servers?  That machine is already
> >> >> > overallocated, by which I mean that the sum of the max heap sizes
of all
> >> >> > java processes running there is greater than the amount physical
memory,
> >> >> > which can lead to swapping.  We have: Hadoop data node, Hadoop
task
> >> >> > tracker, ZooKeeper peer, and region server.  The machine has 8G
of
> >> >> > physical memory.  The region server currently has a max heap size
of 4G.
> >> >> > Should I increase to 6G?  Should I decrease the block cache back
down to
> >> >> > 20% or even lower?  Do we need to move to a 16G server?
> >> >> >
> >> >> > Thanks,
> >> >> > James
> >> >> >
> >> >> >
> >> >> > On Tue, 2010-02-16 at 00:48 -0600, Dan Washusen wrote:
> >> >> > > 32% IO on region server 3!  Ouch! :)
> >> >> > >
> >> >> > > Increasing the block cache to 40% of VM memory without upping
the total
> >> >> > > available memory may only exacerbated the issue.  I notice
that region
> >> >> > > server 2 was already using 3300mb of the 4000mb heap. By
increasing the
> >> >> > > block cache size to 40% you have now given the block cache
1600mb
> >> >> > compared
> >> >> > > to the previous 800mb...
> >> >> > >
> >> >> > > Can you give the region servers more memory?
> >> >> > >
> >> >> > > Cheers,
> >> >> > > Dan
> >> >> > >
> >> >> > > On 16 February 2010 17:42, James Baldassari <james@dataxu.com>
wrote:
> >> >> > >
> >> >> > > > On Tue, 2010-02-16 at 00:14 -0600, Stack wrote:
> >> >> > > > > On Mon, Feb 15, 2010 at 10:05 PM, James Baldassari
<james@dataxu.com
> >> >> > >
> >> >> > > > wrote:
> >> >> > > > > >  Applying HBASE-2180 isn't really an option
at this
> >> >> > > > > > time because we've been told to stick with
the Cloudera distro.
> >> >> > > > >
> >> >> > > > > I'm sure the wouldn't mind (smile).  Seems to about
double
> >> >> > throughput.
> >> >> > > >
> >> >> > > > Hmm, well I might be able to convince them ;)
> >> >> > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > > If I had to guess, I would say the performance
issues start to
> >> >> > happen
> >> >> > > > > > around the time the region servers hit max
heap size, which occurs
> >> >> > > > > > within minutes of exposing the app to live
traffic.  Could GC be
> >> >> > > > killing
> >> >> > > > > > us?  We use the concurrent collector as suggested.
 I saw on the
> >> >> > > > > > performance page some mention of limiting
the size of the new
> >> >> > > > generation
> >> >> > > > > > like -XX:NewSize=6m -XX:MaxNewSize=6m.  Is
that worth trying?
> >> >> > > > >
> >> >> > > > > Enable GC logging for a while?  See hbase-env.sh.
 Uncomment this
> >> >> > line:
> >> >> > > > >
> >> >> > > > > # export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails
> >> >> > > > > XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log"
> >> >> > > >
> >> >> > > > I did uncomment that line, but I can't figure out where
the
> >> >> > gc-hbase.log
> >> >> > > > is.  It's not with the other logs.  When starting HBase
the GC output
> >> >> > > > seems to be going to stdout rather than the file.  Maybe
a Cloudera
> >> >> > > > thing.  I'll do some digging.
> >> >> > > >
> >> >> > > > >
> >> >> > > > > You are using recent JVM?  1.6.0_10 or greater?
 1.6.0_18 might have
> >> >> > > > issues.
> >> >> > > >
> >> >> > > > We're on 1.6.0_16 at the moment.
> >> >> > > >
> >> >> > > > >
> >> >> > > > > Whats CPU and iowait or wa in top look like on
these machines,
> >> >> > > > > particularly the loaded machine?
> >> >> > > > >
> >> >> > > > > How many disks in the machines?
> >> >> > > >
> >> >> > > > I'll have to ask our ops guys about the disks.  The
high load has now
> >> >> > > > switched from region server 1 to 3.  I just saw in our
logs that it
> >> >> > took
> >> >> > > > 139383.065 milliseconds to do 5000 gets, ~36 gets/second,
ouch.  Here
> >> >> > > > are the highlights from top for each region server:
> >> >> > > >
> >> >> > > > Region Server 1:
> >> >> > > > top - 01:39:41 up 4 days, 13:44,  4 users,  load average:
1.89, 0.99,
> >> >> > 1.19
> >> >> > > > Tasks: 194 total,   1 running, 193 sleeping,   0 stopped,
  0 zombie
> >> >> > > > Cpu(s): 15.6%us,  5.8%sy,  0.0%ni, 76.9%id,  0.0%wa,
 0.1%hi,  1.6%si,
> >> >> > > >  0.0%st
> >> >> > > > Mem:   8166588k total,  8112812k used,    53776k free,
    8832k
> >> >> > buffers
> >> >> > > > Swap:  1052248k total,      152k used,  1052096k free,
 2831076k cached
> >> >> > > >  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM  
 TIME+  COMMAND
> >> >> > > > 21961 hadoop    19   0 4830m 4.2g  10m S 114.3 53.6
 37:26.58 java
> >> >> > > > 21618 hadoop    21   0 4643m 578m 9804 S 66.1  7.3 
19:06.89 java
> >> >> > > >
> >> >> > > > Region Server 2:
> >> >> > > > top - 01:40:28 up 4 days, 13:43,  4 users,  load average:
3.93, 2.17,
> >> >> > 1.39
> >> >> > > > Tasks: 194 total,   1 running, 193 sleeping,   0 stopped,
  0 zombie
> >> >> > > > Cpu(s): 11.3%us,  3.1%sy,  0.0%ni, 83.4%id,  1.2%wa,
 0.1%hi,  0.9%si,
> >> >> > > >  0.0%st
> >> >> > > > Mem:   8166588k total,  7971572k used,   195016k free,
   34972k
> >> >> > buffers
> >> >> > > > Swap:  1052248k total,      152k used,  1052096k free,
 2944712k cached
> >> >> > > >  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM  
 TIME+  COMMAND
> >> >> > > > 15752 hadoop    18   0 4742m 4.1g  10m S 210.6 53.1
 41:52.80 java
> >> >> > > > 15405 hadoop    20   0 4660m 317m 9800 S 114.0  4.0
 27:34.17 java
> >> >> > > >
> >> >> > > > Region Server 3:
> >> >> > > > top - 01:40:35 up 2 days,  9:04,  4 users,  load average:
10.15, 11.05,
> >> >> > > > 11.79
> >> >> > > > Tasks: 195 total,   1 running, 194 sleeping,   0 stopped,
  0 zombie
> >> >> > > > Cpu(s): 28.7%us, 10.1%sy,  0.0%ni, 25.8%id, 32.9%wa,
 0.1%hi,  2.4%si,
> >> >> > > >  0.0%st
> >> >> > > > Mem:   8166572k total,  8118592k used,    47980k free,
    3264k
> >> >> > buffers
> >> >> > > > Swap:  1052248k total,      140k used,  1052108k free,
 2099896k cached
> >> >> > > >  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM  
 TIME+  COMMAND
> >> >> > > > 15636 hadoop    18   0 4806m 4.2g  10m S 206.9 53.3
 87:48.81 java
> >> >> > > > 15243 hadoop    18   0 4734m 1.3g 9800 S 117.6 16.7
 63:46.52 java
> >> >> > > >
> >> >> > > > -James
> >> >> > > >
> >> >> > > > >
> >> >> > > > > St>Ack
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > >
> >> >> > > > > > Here are the new region server stats along
with load averages:
> >> >> > > > > >
> >> >> > > > > > Region Server 1:
> >> >> > > > > > request=0.0, regions=16, stores=16, storefiles=33,
> >> >> > > > storefileIndexSize=4, memstoreSize=1, compactionQueueSize=0,
> >> >> > usedHeap=2891,
> >> >> > > > maxHeap=4079, blockCacheSize=1403878072, blockCacheFree=307135816,
> >> >> > > > blockCacheCount=21107, blockCacheHitRatio=84, fsReadLatency=0,
> >> >> > > > fsWriteLatency=0, fsSyncLatency=0
> >> >> > > > > > Load Averages: 10.34, 10.58, 7.08
> >> >> > > > > >
> >> >> > > > > > Region Server 2:
> >> >> > > > > > request=0.0, regions=15, stores=16, storefiles=26,
> >> >> > > > storefileIndexSize=3, memstoreSize=1, compactionQueueSize=0,
> >> >> > usedHeap=3257,
> >> >> > > > maxHeap=4079, blockCacheSize=661765368, blockCacheFree=193741576,
> >> >> > > > blockCacheCount=9942, blockCacheHitRatio=77, fsReadLatency=0,
> >> >> > > > fsWriteLatency=0, fsSyncLatency=0
> >> >> > > > > > Load Averages: 1.90, 1.23, 0.98
> >> >> > > > > >
> >> >> > > > > > Region Server 3:
> >> >> > > > > > request=0.0, regions=16, stores=16, storefiles=41,
> >> >> > > > storefileIndexSize=4, memstoreSize=4, compactionQueueSize=0,
> >> >> > usedHeap=1627,
> >> >> > > > maxHeap=4079, blockCacheSize=665117184, blockCacheFree=190389760,
> >> >> > > > blockCacheCount=9995, blockCacheHitRatio=70, fsReadLatency=0,
> >> >> > > > fsWriteLatency=0, fsSyncLatency=0
> >> >> > > > > > Load Averages: 2.01, 3.56, 4.18
> >> >> > > > > >
> >> >> > > > > > That first region server is getting hit much
harder than the
> >> >> > others.
> >> >> > > > > > They're identical machines (8-core), and the
distribution of keys
> >> >> > > > should
> >> >> > > > > > be fairly random, so I'm not sure why that
would happen.  Any other
> >> >> > > > > > ideas or suggestions would be greatly appreciated.
> >> >> > > > > >
> >> >> > > > > > Thanks,
> >> >> > > > > > James
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > > > On Mon, 2010-02-15 at 21:51 -0600, Stack wrote:
> >> >> > > > > >> Yeah, I was going to say that if your
loading is mostly read, you
> >> >> > can
> >> >> > > > > >> probably go up from the 0.2 given over
to cache.  I like Dan's
> >> >> > > > > >> suggestion of trying it first on one server,
if you can.
> >> >> > > > > >>
> >> >> > > > > >> St.Ack
> >> >> > > > > >>
> >> >> > > > > >> On Mon, Feb 15, 2010 at 5:22 PM, Dan Washusen
<dan@reactive.org>
> >> >> > > > wrote:
> >> >> > > > > >> > So roughly 72% of reads use the blocks
held in the block
> >> >> > cache...
> >> >> > > > > >> >
> >> >> > > > > >> > It would be interesting to see the
difference between when it
> >> >> > was
> >> >> > > > working OK
> >> >> > > > > >> > and now.  Could you try increasing
the memory allocated to one
> >> >> > of
> >> >> > > > the
> >> >> > > > > >> > regions and also increasing the "hfile.block.cache.size"
to say
> >> >> > > > '0.4' on the
> >> >> > > > > >> > same region?
> >> >> > > > > >> >
> >> >> > > > > >> > On 16 February 2010 11:54, James
Baldassari <james@dataxu.com>
> >> >> > > > wrote:
> >> >> > > > > >> >
> >> >> > > > > >> >> Hi Dan.  Thanks for your suggestions.
 I am doing writes at the
> >> >> > > > same
> >> >> > > > > >> >> time as reads, but there are
usually many more reads than
> >> >> > writes.
> >> >> > > >  Here
> >> >> > > > > >> >> are the stats for all three region
servers:
> >> >> > > > > >> >>
> >> >> > > > > >> >> Region Server 1:
> >> >> > > > > >> >> request=0.0, regions=15, stores=16,
storefiles=34,
> >> >> > > > storefileIndexSize=3,
> >> >> > > > > >> >> memstoreSize=308, compactionQueueSize=0,
usedHeap=3096,
> >> >> > > > maxHeap=4079,
> >> >> > > > > >> >> blockCacheSize=705474544, blockCacheFree=150032400,
> >> >> > > > blockCacheCount=10606,
> >> >> > > > > >> >> blockCacheHitRatio=76, fsReadLatency=0,
fsWriteLatency=0,
> >> >> > > > fsSyncLatency=0
> >> >> > > > > >> >>
> >> >> > > > > >> >> Region Server 2:
> >> >> > > > > >> >> request=0.0, regions=16, stores=16,
storefiles=39,
> >> >> > > > storefileIndexSize=4,
> >> >> > > > > >> >> memstoreSize=225, compactionQueueSize=0,
usedHeap=3380,
> >> >> > > > maxHeap=4079,
> >> >> > > > > >> >> blockCacheSize=643172800, blockCacheFree=212334144,
> >> >> > > > blockCacheCount=9660,
> >> >> > > > > >> >> blockCacheHitRatio=69, fsReadLatency=0,
fsWriteLatency=0,
> >> >> > > > fsSyncLatency=0
> >> >> > > > > >> >>
> >> >> > > > > >> >> Region Server 3:
> >> >> > > > > >> >> request=0.0, regions=13, stores=13,
storefiles=31,
> >> >> > > > storefileIndexSize=4,
> >> >> > > > > >> >> memstoreSize=177, compactionQueueSize=0,
usedHeap=1905,
> >> >> > > > maxHeap=4079,
> >> >> > > > > >> >> blockCacheSize=682848608, blockCacheFree=172658336,
> >> >> > > > blockCacheCount=10262,
> >> >> > > > > >> >> blockCacheHitRatio=72, fsReadLatency=0,
fsWriteLatency=0,
> >> >> > > > fsSyncLatency=0
> >> >> > > > > >> >>
> >> >> > > > > >> >> The average blockCacheHitRatio
is about 72.  Is this too low?
> >> >> > > >  Anything
> >> >> > > > > >> >> else I can check?
> >> >> > > > > >> >>
> >> >> > > > > >> >> -James
> >> >> > > > > >> >>
> >> >> > > > > >> >>
> >> >> > > > > >> >> On Mon, 2010-02-15 at 18:16 -0600,
Dan Washusen wrote:
> >> >> > > > > >> >> > Maybe the block cache is
thrashing?
> >> >> > > > > >> >> >
> >> >> > > > > >> >> > If you are regularly writing
data to your tables then it's
> >> >> > > > possible that
> >> >> > > > > >> >> the
> >> >> > > > > >> >> > block cache is no longer
being effective.  On the region
> >> >> > server
> >> >> > > > web UI
> >> >> > > > > >> >> check
> >> >> > > > > >> >> > the blockCacheHitRatio value.
 You want this value to be high
> >> >> > (0
> >> >> > > > - 100).
> >> >> > > > > >> >>  If
> >> >> > > > > >> >> > this value is low it means
that HBase has to go to disk to
> >> >> > fetch
> >> >> > > > blocks
> >> >> > > > > >> >> of
> >> >> > > > > >> >> > data.  You can control the
amount of VM memory that HBase
> >> >> > > > allocates to
> >> >> > > > > >> >> the
> >> >> > > > > >> >> > block cache using the "hfile.block.cache.size"
property
> >> >> > (default
> >> >> > > > is 0.2
> >> >> > > > > >> >> > (20%)).
> >> >> > > > > >> >> >
> >> >> > > > > >> >> > Cheers,
> >> >> > > > > >> >> > Dan
> >> >> > > > > >> >> >
> >> >> > > > > >> >> > On 16 February 2010 10:45,
James Baldassari <
> >> >> > james@dataxu.com>
> >> >> > > > wrote:
> >> >> > > > > >> >> >
> >> >> > > > > >> >> > > Hi,
> >> >> > > > > >> >> > >
> >> >> > > > > >> >> > > Does anyone have any
tips to share regarding optimization
> >> >> > for
> >> >> > > > random
> >> >> > > > > >> >> > > read performance? 
For writes I've found that setting a
> >> >> > large
> >> >> > > > write
> >> >> > > > > >> >> > > buffer and setting
auto-flush to false on the client side
> >> >> > > > significantly
> >> >> > > > > >> >> > > improved put performance.
 Are there any similar easy
> >> >> > tweaks to
> >> >> > > > improve
> >> >> > > > > >> >> > > random read performance?
> >> >> > > > > >> >> > >
> >> >> > > > > >> >> > > I'm using HBase 0.20.3
in a very read-heavy real-time
> >> >> > system
> >> >> > > > with 1
> >> >> > > > > >> >> > > master and 3 region
servers.  It was working ok for a
> >> >> > while,
> >> >> > > > but today
> >> >> > > > > >> >> > > there was a severe
degradation in read performance.
> >> >> >  Restarting
> >> >> > > > Hadoop
> >> >> > > > > >> >> > > and HBase didn't help,
are there are no errors in the logs.
> >> >> > > >  Read
> >> >> > > > > >> >> > > performance starts
off around 1,000-2,000 gets/second but
> >> >> > > > quickly
> >> >> > > > > >> >> > > (within minutes) drops
to around 100 gets/second.
> >> >> > > > > >> >> > >
> >> >> > > > > >> >> > > I've already looked
at the performance tuning wiki page.
> >> >> >  On
> >> >> > > > the server
> >> >> > > > > >> >> > > side I've increased
hbase.regionserver.handler.count from
> >> >> > 10 to
> >> >> > > > 100,
> >> >> > > > > >> >> but
> >> >> > > > > >> >> > > it didn't help.  Maybe
this is expected because I'm only
> >> >> > using
> >> >> > > > a single
> >> >> > > > > >> >> > > client to do reads.
 I'm working on implementing a client
> >> >> > pool
> >> >> > > > now, but
> >> >> > > > > >> >> > > I'm wondering if there
are any other settings on the server
> >> >> > or
> >> >> > > > client
> >> >> > > > > >> >> > > side that might improve
things.
> >> >> > > > > >> >> > >
> >> >> > > > > >> >> > > Thanks,
> >> >> > > > > >> >> > > James
> >> >> > > > > >> >> > >
> >> >> > > > > >> >> > >
> >> >> > > > > >> >> > >
> >> >> > > > > >> >>
> >> >> > > > > >> >>
> >> >> > > > > >> >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > >
> >> >> > > >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >


Mime
View raw message