hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Appy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16524) Procedure v2 - Compute WALs cleanup on wal modification and not on every sync
Date Wed, 04 Jan 2017 14:49:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798436#comment-15798436

Appy commented on HBASE-16524:

Reviewed it just now. Looks good to me.
Should probably change the functions {{insert(BitSetNode node, long procId)}} and others to
{{insert(long procId, BitSetNode nodeHint)}} because the first argument isn't mandatory, instead
it's a hint to avoid lookup into map repeatedly.

One thing worth mentioning, wal cleanup logic gets triggered on every wal sync, (WalProcedureStore#syncLoop),
so the effort here is to make it quick.
One case here is, if a procedure gets stuck and was last synced a while back, we keep holding
intermediate wals. Once the stuck procedure finishes, it'll take some time to cleanup wals
since each wal will be sequentially processed as 'holding' wall. This will delay the next
sync a bit.
Just a heads up in case we see issues related to wal.

> Procedure v2 - Compute WALs cleanup on wal modification and not on every sync
> -----------------------------------------------------------------------------
>                 Key: HBASE-16524
>                 URL: https://issues.apache.org/jira/browse/HBASE-16524
>             Project: HBase
>          Issue Type: Sub-task
>          Components: proc-v2
>    Affects Versions: 2.0.0
>            Reporter: Appy
>            Assignee: Matteo Bertozzi
>            Priority: Minor
>             Fix For: 2.0.0
>         Attachments: HBASE-16524-v2.patch, HBASE-16524-v3.patch, HBASE-16524-v4.patch,
HBASE-16524-v5.patch, HBASE-16524-v6.patch, HBASE-16524.master.001.patch, HBASE-16524.master.002.patch,
> Fix performance regression introduced by HBASE-16094.
> Instead of scanning all the wals every time, we can rely on the insert/update/delete
events we have.
> and since we want to delete the wals in order we can keep track of what is "holding"
that wal, and take a hit on scanning all the trackers only when we remove the first log in
the queue.
> e.g.
> WAL-1 [1, 2] 
> WAL-2 [1] -> "[2] is holding WAL-1"
> WAL-3 [2] -> "WAL 1 can be removed, recompute what is holding WAL-2"

This message was sent by Atlassian JIRA

View raw message