hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From y_823...@tsmc.com
Subject RE: HBase reading performance
Date Fri, 26 Feb 2010 09:54:39 GMT
My META table just be splitted by one region in one machine.
Name Region Server Encoded Name Start Key End Key
.META.,,1 f12p4cc11.tsmc:60030 -
That machine has high CPU and network usage.
I guess that's the bottleneck of multiple client reading performance.
Any ideas ?
Thanks

Fleming Chiu(邱宏明)
707-6128
y_823910@tsmc.com
週一無肉日吃素救地球(Meat Free Monday Taiwan)




                                                                                         
                                                            
                      y_823910@tsmc.com                                                  
                                                            
                                               To:      hbase-user@hadoop.apache.org     
                                                            
                      2010/02/26 08:42         cc:      (bcc: Y_823910/TSMC)             
                                                            
                      AM                       Subject: RE: HBase reading performance    
                                                            
                      Please respond to                                                  
                                                            
                      hbase-user                                                         
                                                            
                                                                                         
                                                            
                                                                                         
                                                            




Sorry for my ignoring their basic informations.

What kind of reading are you doing ? Random reads or scans?  What
version of hbase?
---------------------------------------------------------------------------------------


HBase Version 0.20.2, r834515

ResultScanner scanner = table2.getIndexedScanner(idx_column,
            Bytes.toBytes(idx_value), Bytes.toBytes(idx_value+"~"),null,
filter,
            new byte[][] { Bytes.toBytes(columnfamily+":")});


Are you using the native APIs? Stargate? something else?
----------------------------------------------------------------------------------------

Yes, I am using the native APIS.
I dispatched these java code to different machines and run it concurrently
through GridGain.

----------------------------------------------------------------------------------------


My zookeeper servers are not in the dedicated machine.
<name>hbase.zookeeper.property.maxClientCnxns</name>
    <value>1000</value>
----------------------------------------------------------------------------------------


Region Count
f12p4cc06.tsmc:60030 1266903209803 requests=0, regions=143, usedHeap=1396,
maxHeap=1991
f12p4cc07.tsmc:60030 1266903219641 requests=193, regions=147, usedHeap=897,
maxHeap=1991
f12p4cc08.tsmc:60030 1266903209410 requests=0, regions=151, usedHeap=955,
maxHeap=1991
f12p4cc09.tsmc:60030 1266903210645 requests=0, regions=153, usedHeap=1162,
maxHeap=1991
f12p4cc10.tsmc:60030 1266903208231 requests=0, regions=155, usedHeap=1102,
maxHeap=1991
f12p4cc11.tsmc:60030 1266903208718 requests=0, regions=146, usedHeap=1321,
maxHeap=1991
f12p4cc12.tsmc:60030 1266903205539 requests=0, regions=142, usedHeap=659,
maxHeap=1991
f12p4cc13.tsmc:60030 1266903214999 requests=0, regions=157, usedHeap=1528,
maxHeap=1991
f12p4cc14.tsmc:60030 1266903215714 requests=0, regions=153, usedHeap=939,
maxHeap=1991
f12p4cc15.tsmc:60030 1266903207766 requests=0, regions=153, usedHeap=1004,
maxHeap=1991
Total:  servers: 10   requests=193, regions=1500



Fleming Chiu(邱宏明)
707-6128
y_823910@tsmc.com
週一無肉日吃素救地球(Meat Free Monday Taiwan)





                      Michael Segel

                      <michael_segel@ho        To:
<hbase-user@hadoop.apache.org>

                      tmail.com>               cc:      (bcc:
Y_823910/TSMC)

                                               Subject: RE: HBase reading
performance

                      2010/02/25 10:10

                      PM

                      Please respond to

                      hbase-user








Also how are you reading from HBase?
Are you using the native APIs? Stargate? something else?


> Date: Thu, 25 Feb 2010 04:51:39 -0800
> Subject: Re: HBase reading performance
> From: stack@duboce.net
> To: hbase-user@hadoop.apache.org
>
> What kind of reading are you doing ? Random reads or scans?  What
> version of hbase?
> Thanks,
> St.Ack
>
> 2010/2/25  <y_823910@tsmc.com>:
> > Hi,
> >
> > We have 20 1U servers (4 core, 12G ram) as a cluster,
> > 3 zookeepers, 10 region servers.
> > My program is to read hbase table data one by one.
> >    Read -> hbaseTable1
> >            hbaseTable2
> >            hbaseTable3
> >                 .
> >                 .
> >                 .
> >            hbaseTable30
> > ---------------------------------------------------
> > I collected some performance data while I started different number of
> > client to read data
> > from hbase concurrently.
> >
> > 10  client duration :    53  (sec)
> > 50                      181
> > 200                     839
> > 400                    1801
> >
> > I think the multiple reading performance is not good.
> > How can I start to fine tune it?
> > Any suggestion would be appreciated!
> >
> >
> > Fleming Chiu(邱宏明)
> > 707-6128
> > y_823910@tsmc.com
> > 週一無肉日吃素救地球(Meat Free Monday Taiwan)
> >
> >
> >
---------------------------------------------------------------------------
> >                                                         TSMC PROPERTY
> >  This email communication (and any attachments) is proprietary
information
> >  for the sole use of its
> >  intended recipient. Any unauthorized review, use or distribution by
anyone
> >  other than the intended
> >  recipient is strictly prohibited.  If you are not the intended
recipient,
> >  please notify the sender by
> >  replying to this email, and then delete this email and any copies of
it
> >  immediately. Thank you.
> >
---------------------------------------------------------------------------
> >
> >
> >
> >

_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
http://clk.atdmt.com/GBL/go/201469229/direct/01/




---------------------------------------------------------------------------

                                                         TSMC PROPERTY

 This email communication (and any attachments) is proprietary information

 for the sole use of its

 intended recipient. Any unauthorized review, use or distribution by anyone

 other than the intended
 recipient is strictly prohibited.  If you are not the intended recipient,

 please notify the sender by

 replying to this email, and then delete this email and any copies of it

 immediately. Thank you.


---------------------------------------------------------------------------








 --------------------------------------------------------------------------- 
                                                         TSMC PROPERTY       
 This email communication (and any attachments) is proprietary information   
 for the sole use of its                                                     
 intended recipient. Any unauthorized review, use or distribution by anyone  
 other than the intended                                                     
 recipient is strictly prohibited.  If you are not the intended recipient,   
 please notify the sender by                                                 
 replying to this email, and then delete this email and any copies of it     
 immediately. Thank you.                                                     
 --------------------------------------------------------------------------- 




Mime
View raw message