Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Message-ID: <2112875521.1261675589433.JavaMail.jira@brutus>
Date: Thu, 24 Dec 2009 17:26:29 +0000 (UTC)
From: "Billy Pearson (JIRA)" <jira@apache.org>
To: hbase-dev@hadoop.apache.org
Subject: [jira] Commented: (HBASE-2053) Upper bound of outstanding WALs can
 be overrun
In-Reply-To: <1707219796.1261076058127.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794462#action_12794462 ] 

Billy Pearson commented on HBASE-2053:
--------------------------------------

I have a table that runs on 7 servers and it is 714 regions with a split size of 512mb so all quite large
I set the block size to 32MB so the index size is vary low < 1mb on each server so each server can handle a lot of regions.
I have test and one region server at one points could handle ~680 regions with out memory problems heap is 1gb.
I can do this because I do not need random access on this table form my apps so only full scans the table.

The problem is coming from hosting ~120 regions on each server as edits come in the hlogs roll and flushes happen on the hot regions with the largest memstore the smaller memstores only get flushed when the hlog limit is reached and it starts flushing to help clear out the old logs. But this only happens once per hlog roll would could have 100 region edits or more outstanding edits in this hlog. 

I set my max logs to 24 and I see 100-120 log files a lot when data is getting added.

I thank a simple fix for this would be to flushed the oldest outstanding seqnum and loop or queue flushes until it flushes all outstanding memstore edits for the oldest log file.
I am just not sure how that would work with the code for the next hlog roll if we are still flushing from the first one we do not want t block a hlog roll.

My only problem with this is when I have a region fail for some reason the master may have to process 100+ logs to get going again
I know I read some where that we had plans on moving the recovering of logs from the master the the region servers so it could happen much more quickly.

Also note we might have to start dealing with more hlogs hanging around with replication in the works.
So in the end this may not be a issue that we need to fix I do not know what the memory impact would be with 100's of logs 


> Upper bound of outstanding WALs can be overrun
> ----------------------------------------------
>
>                 Key: HBASE-2053
>                 URL: https://issues.apache.org/jira/browse/HBASE-2053
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>
> Kevin Peterson up on hbase-user posted the following.  Of interest is the link on the end which is logs of WAL rolls and removals.  In once place we remove 70plus logs because the outstanding edits have moved passed the outstanding sequence numbers -- so our basic WAL removal mechanism is working -- but if you study the log, the tendency is steady climb in the number of logs.   HLog#cleanOldLogs needs to notice such an upward tendency and work more aggressively cleaning the old in this case.  Here is Kevin's note:
> {code}
> n Tue, Dec 15, 2009 at 3:17 PM, Kevin Peterson <x@y.com> wrote:
> This makes some sense now. I currently have 2200 regions across 3 tables. My
> largest table accounts for about 1600 of those regions and is mostly active
> at one end of the keyspace -- our key is based on date, but data only
> roughly arrives in order. I also write to two secondary indexes, which have
> no pattern to the key at all. One of these secondary tables has 488 regions
> and the other has 96 regions.
> We write about 10M items per day to the main table (articles). All of these
> get written to one of the secondary indexes (article-ids). About a third get
> written to the other secondary index. Total volume of data is about 10GB /
> day written.
> I think the key is as you say that the regions aren't filled enough to
> flush. The articles table gets mostly written to near one end and I see
> splits happening regularly. The index tables have no pattern so the 10
> millions writes get scattered across the different regions. I've looked more
> closely at a log file (linked below), and if I forget about my main table
> (which would tend to get flushed), and look only at the indexes, this seems
> to be what's happening:
> 1. Up to maxLogs HLogs, it doesn't do any flushes.
> 2. Once it gets above maxLogs, it will start flushing one region each time
> it creates a new HLog.
> 3. If the first HLog had edits for say 50 regions, it will need to flush the
> region with oldest edits 50 times before the HLog can be removed.
> If N is the number of regions getting written to, but not getting enough
> writes to flush on their own, then I think this converges to maxLogs + N
> logs on average. If I think of maxLogs as "number of logs to start flushing
> regions at" this makes sense.
> http://kdpeterson.net/paste/hbase-hadoop-regionserver-mi-prod-app35.ec2.biz360.com.log.2009-12-14
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.