hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LN (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-745) scaling of one regionserver, improving memory and cpu usage
Date Tue, 15 Jul 2008 07:59:31 GMT
scaling of one regionserver, improving memory and cpu usage
-----------------------------------------------------------

                 Key: HBASE-745
                 URL: https://issues.apache.org/jira/browse/HBASE-745
             Project: Hadoop HBase
          Issue Type: Improvement
          Components: regionserver
    Affects Versions: 0.1.3
         Environment: hadoop 0.17.1
            Reporter: LN
            Priority: Minor


after weeks testing hbase 0.1.3 and hadoop(0.16.4, 0.17.1), i found there are many works to
do,  before a particular regionserver can handle data about 100G, or even more. i'd share
my opions here with stack, and other developers.

first, the easiest way improving scalability of regionserver is upgrading hardware, use 64bit
os and 8G memory for the regionserver process, and speed up disk io. 

besides hardware, following are software bottlenecks i found in regionserver:
1. as data increasing, compaction was eating cpu(with io) times, the total compaction time
is basicly linear relative to whole data size, even worse, sometimes square relavtive to that
size.
2. memory and socket connection usage are depends on opened mapfiles, see HADOOP-2341 and
HBASE-24. 

will explain above in comments later.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message