hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wayne <wav...@gmail.com>
Subject Re: Breaking down an HBase read through thrift
Date Mon, 10 Jan 2011 19:07:03 GMT
Thank you for the help. Below are a few more questions.

On Mon, Jan 10, 2011 at 1:41 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> Inline.
>
> > *Region/Meta Cache
> > *
> > Often times the region list is not "hot" and thrift has to talk to the
> meta
> > table. We have 6k+ regions and growing quickly and expect 1k+/node. Can
> we
> > help our performance by pre-caching all region locations? How many
> regions
> > can thrift keep before over-writing in cache (given default settings)?
> Does
> > it make sense to write a program to cycle through all thrift servers and
> > heat up the region cache after a lot of writes?
>
> You may want to keep the size of your regions bigger, to have fewer of
> them. Checkout the MAX_FILESIZE config in the shel when
> creating/altering a table.
>

Is this the same as the hregion.max.filesize setting?

>
> BTW the Thrift server is really just an interface between incoming
> thrift calls and the HBase java API. When you say "How many regions
> can thrift keep before over-writing in cache", it's actually the hbase
> client in HConnectionManager that does it. And the answer to that
> question is... well it depends on the memory you give to the thrift
> servers because the references are "weak", meaning that the GC will
> decide whether to collect some of them if there's memory pressure.
>

If HConnectionManager is doing it is it then hbase memory of Thrift server
memory that caches the region locations?

>
> Regarding pre-fetching of region locations, it's already configured to
> fetch 10 consecutive rows every time the client has to query the
> .META. table, you can change this with hbase.client.prefetch.limit
>

I will try this. We have sparse data but maybe setting very high will take
the hit up front. Is this setting only used when a region location is not in
cache? If we set this very high we do not want to take the hit on every
read.

>
> >
> > *Thrift Logs
> > *Below is an example of a scan from the thrift logs. I am trying to
> > understand what is going on and where things can be sped up. The
> > scannerGetList log below took 43ms which is longer than usual. What would
> > slow this down, waiting for the region server to respond (there is load
> on
> > the cluster for writes)? It took 12ms to be finished with scanning which
> > makes sense, but then it appears to take almost as long to close the
> scanner
> > (11ms). This read took 66ms when in reality it only took 12ms to read the
> > data from what I can tell. That does not make sense to me. The disk i/o
> > should be the bulk of the time spent on a read and it appears to be
> little
> > of this 66ms. How can we speed up read latency? Faster disks should be
> the
> > answer but I am not sure it makes much of a difference here.
> >
> > 2011-01-08 16:28:47,921 DEBUG
> > org.apache.hadoop.hbase.client.HTable$ClientScanner: Creating scanner
> over
> > xxx starting at key '12345'
> > 2011-01-08 16:28:47,921 DEBUG
> > org.apache.hadoop.hbase.client.HTable$ClientScanner: Advancing internal
> > scanner to startKey at '12345'
> > 2011-01-08 16:28:47,964 DEBUG
> > org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler: scannerGetList:
> > id=145060
> > 2011-01-08 16:28:47,976 DEBUG
> > org.apache.hadoop.hbase.client.HTable$ClientScanner: Finished with
> scanning
> > at REGION => {NAME => ....
> > 2011-01-08 16:28:47,987 DEBUG
> > org.apache.hadoop.hbase.thrift.ThriftServer$HBaseHandler: scannerClose:
> > id=145060
>
> What happened here is 3 RPCs: open the scanner, get, and finally
> close. Ideally the first two should be done in the same RPC, it
> shouldn't even be that hard to implement just for the thrift server,
> and then the close isn't required if you provide is stopRow. It seems
> in your case a lot of time was spent doing those RPCs, but better
> instrumenting would be required to tell whether the 43ms was really
> spent on the network or opening the scanner. Have a look at
> ThriftServer, it's pretty straight forward.
>

We have enough issues with trying to get hbase to run...don't have the
resources to crack open the thrift code at this point. I appreciate the tips
on what can be done.

>
> Regarding the actual reading of the data, if you plan on always
> scanning a lot of rows at the same time, I suggest setting
> hbase.client.scanner.caching to something bigger than 1. I know it's
> dumb to have to restart all the thrift servers in order to change
> configuration and that all scans will have the same caching value but
> until Scan.setCaching() is exposed to the Thrift client, it has to be
> set that way.
>

We are using the client.scannerGetList(scanID,scanBatchSize) with a higher
batch size. That provides the same functionality correct? We have some very
large rows we definitely do not want to get 50 at a time. Our biggest read
performance concern though is small rows we get 25 at once
(scanBatchSize=25).


>
>
> J-D
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message