hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lohit <lohit.vijayar...@gmail.com>
Subject Re: Random Reads throughput/performance
Date Fri, 24 Jun 2011 20:41:05 GMT
2011/6/23 Sateesh Lakkarsu <lakkarsu@gmail.com>

> We have been testing random reads and from a 6 node cluster (1NN, 5DN, 1HM,
> 5RS each with 48G, 5 disks) right now seeing a throughput of 1100 per sec
> per node. Most of the configs are default, except 4G for RS, *handler.count
> and gc (
>
> http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/
> ).
> The actual data is 300 byte records, each of which can have upwards of 1000
> versions keyed on id which is random/uuid based. Using LZO compression.
> There are 300M records with distribution almost even across RS which are
> pre-created and bulk loaded.
>

We have done similar tests by with 1K byte records and 1B rows, without
compression and see peaks of 400 ops/sec.
Considering that, you might be able to push it a little more. How much
memory is given for DataNode and Region server.
Also, is your block cache set to default of 20%?
Can you tell us what numbers do you see without compression.


>
> I have read the below related posts, but want to see if there is any new
> input in regards to:
>
> - improving random read performance in general
>
> - if we have more disks or say double them, can we expect a much higher
> throughput per node? We tried to drive more concurrent reads through
> current
> 5 nodes/5 disks but don't see much change. Looks like RS queue is growing
> and as does CPU usage... anything we can tweak in RS to let more reads go
> through? or is this the best we can get?
>
>
> Thanks.
>
>
>
> Related threads in one place:
>
> http://hbase.apache.org/book.html#performance
>
> http://wiki.apache.org/hadoop/PerformanceTuning
>
> http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation
>
>
> http://search-hadoop.com/m/Umugv1srG951&subj=Re+How+to+improve+random+read+latency+
>
>
> https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/e6024f24a54d6aed
>
> http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/11539
>
> http://www.mail-archive.com/hbase-user@hadoop.apache.org/msg06714.html
>
> http://hstack.org/hbase-performance-testing/
>
>
> http://groups.google.com/group/nosql-databases/browse_thread/thread/dc5b30a6863fe42d/b1d54a5b7f99c3fb
>
> http://comments.gmane.org/gmane.comp.java.hadoop.hbase.user/8653
>
> http://cloudepr.blogspot.com/2009/08/hbase-0200-performance-evaluation.html
>
>
> http://apache-hbase.679495.n3.nabble.com/Re-jira-Created-HBASE-3303-Lower-hbase-regionserver-handler-count-from-25-back-to-10-td2008741.html
>



-- 
Have a Nice Day!
Lohit

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message