hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3325) Optimize log splitter to not output obsolete edits
Date Fri, 10 Dec 2010 02:17:02 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970054#action_12970054
] 

Todd Lipcon commented on HBASE-3325:
------------------------------------

Grabbing seqid from the last flush file seems pretty reasonable too. Stack and I were talking
on IRC and we thought it might be too slow to do that for hundreds of regions for the case
where a server is hosting lots of them.

> Optimize log splitter to not output obsolete edits
> --------------------------------------------------
>
>                 Key: HBASE-3325
>                 URL: https://issues.apache.org/jira/browse/HBASE-3325
>             Project: HBase
>          Issue Type: Improvement
>          Components: master, regionserver
>    Affects Versions: 0.92.0
>            Reporter: Todd Lipcon
>
> Currently when the master splits logs, it outputs all edits it finds, even those that
have already been obsoleted by flushes. At replay time on the RS we discard the edits that
have already been flushed.
> We could do a pretty simple optimization here - basically the RS should replicate a map
"region id -> last flushed seq id" into ZooKeeper (this can be asynchronous by some seconds
without any problems). Then when doing log splitting, if we have this map available, we can
discard any edits found in the logs that were already flushed, and thus output a much smaller
amount of data.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message