hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Kellerman (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1820) [hbase] regionserver creates hlogs without bound
Date Sun, 23 Sep 2007 17:40:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529753
] 

Jim Kellerman commented on HADOOP-1820:
---------------------------------------

There were three HBase tests that failed in build 804:

testLogRolling failed because a region split took an excessive amount of time (43 seconds),
and by the time the split completed, the client had given up and started shutdown of the MiniHBase
cluster. This is undoubtedly caused by the aggressive parameters we set for client retries
(time between retries: 5 seconds, number of retries: 5) to make tests run faster. Because
a region split is highly I/O intensive, the thread will often block on I/O. If the thread
cannot get rescheduled because the client is doing a retry, this could cause the splitting
thread to be starved and cause the long time for the split to complete. By contrast, the test
testBasicSplit which does not have a cluster or a client running, completes a region split
in 1 second.

Similarly a region split in testSplitRegionIsDeleted, which also runs a cluster and a client,
took 50 seconds to complete a region split, and once again the client gave up and started
shutting down the cluster just as the split was completing.

in testTableMapReduce, the default settings for region server timeout are too short so the
master takes the region server out of rotation just before it completed its task.

So it appears that in the build environment, we need to increase the parameters for these
respective tests so that they can run without timing out.

> [hbase] regionserver creates hlogs without bound
> ------------------------------------------------
>
>                 Key: HADOOP-1820
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1820
>             Project: Hadoop
>          Issue Type: Bug
>          Components: contrib/hbase
>    Affects Versions: 0.15.0
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.15.0
>
>         Attachments: excerpt.log, patch.txt, patch.txt, patch.txt, patch.txt
>
>
> Regionserver keeps log of all edits for all the regions its carrying.  Its used recoverying
state if a regionserver crashes: edits that have not been persisted to an HStoreFile are rerun
to populate memcache which in turn is converted to an on-filesytem HStoreFile.  On a period,
the log is rotated and a new one is opened.  While the region server is up, the logs grow
in number without bound.  Only the most recent contain unpersisted edits.  If the region server
goes down clean, then its logs are cleaned up. If a region server crashes, as part of recovery,
the logs of edits are sorted and split  per region.  Recovery would run faster if it did not
have to plough through reams of stale edits.
> Just now, I had a host crash w/ 112 log files each of 30k plus edits each.
> We could rename the log rolling thread the log maintainer.  As well as rolling logs,
it could check for edit logs to clean.  When rolled, logs could be marked with the sequence
id of their last contained edit.  The thread could on a period ask each hosted region for
the "lowest highest" sequence id of all regions deployed.  Once this number had crossed out
that on a particular log, the log could be cleaned up safely.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message