hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Pearson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-745) scaling of one regionserver, improving memory and cpu usage
Date Tue, 22 Jul 2008 07:02:31 GMT

    [ https://issues.apache.org/jira/browse/HBASE-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615538#action_12615538
] 

Billy Pearson commented on HBASE-745:
-------------------------------------

I lost my data I was running to test large import so re downloading I will be able to run
my test on the patch in about 24 hours when I get done processing my dataset again.
Your last post sounds better and more correct if the patch is working correct we pick up efficiency
when we do not have to compact the larger mapfiles with every compaction.
I would assume this will help with the users that still have 32bit server keep the region
server under the 2gb limit by flushing a little more often if needed under load.


> scaling of one regionserver, improving memory and cpu usage
> -----------------------------------------------------------
>
>                 Key: HBASE-745
>                 URL: https://issues.apache.org/jira/browse/HBASE-745
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: regionserver
>    Affects Versions: 0.1.3, 0.2.0
>         Environment: hadoop 0.17.1
>            Reporter: LN
>            Priority: Minor
>         Attachments: hbase-745-for-0.2.patch, HBASE-745.compact.patch
>
>
> after weeks testing hbase 0.1.3 and hadoop(0.16.4, 0.17.1), i found there are many works
to do,  before a particular regionserver can handle data about 100G, or even more. i'd share
my opions here with stack, and other developers.
> first, the easiest way improving scalability of regionserver is upgrading hardware, use
64bit os and 8G memory for the regionserver process, and speed up disk io. 
> besides hardware, following are software bottlenecks i found in regionserver:
> 1. as data increasing, compaction was eating cpu(with io) times, the total compaction
time is basicly linear relative to whole data size, even worse, sometimes square relavtive
to that size.
> 2. memory usage are depends on opened mapfiles
> 3. network connection are depends on opened mapfiles, see HADOOP-2341 and HBASE-24. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message