hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16689) Durability == ASYNC_WAL means no SYNC
Date Fri, 23 Sep 2016 05:59:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515511#comment-15515511
] 

Yu Li commented on HBASE-16689:
-------------------------------

+1, we also found this during benchmark testing weeks ago but didn't find a good way to fix
it, and since there will always be some SYNC call online, we mark this as relatively low priority
on our side. Already got some idea about how to fix this sir? [~stack]

btw, we also found that because currently we are using one disruptor per WAL and the disruptor
is consumed in sequential, we've seen more contention among different regions under high write
pressure. I got a patch at hand and will open another JIRA to tell more details soon, JFYI.

> Durability == ASYNC_WAL means no SYNC
> -------------------------------------
>
>                 Key: HBASE-16689
>                 URL: https://issues.apache.org/jira/browse/HBASE-16689
>             Project: HBase
>          Issue Type: Bug
>          Components: wal
>    Affects Versions: 1.0.3, 1.1.6, 1.2.3
>            Reporter: stack
>            Assignee: stack
>            Priority: Critical
>
> Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table Table appends.
If all tables on a cluster have this setting, data is flushed from the RS to the DN at some
arbitrary time and a bunch may just hang out in DFSClient buffers on the RS-side indefinitely
if writes are sporadic, at least until there is a WAL roll -- a log roll sends a sync through
the write pipeline to flush out any outstanding appends -- or a region close which does similar....
or we crash and drop the data in buffers RS.
> This is probably not what a user expects when they set ASYNC_WAL (We don't doc anywhere
that I could find clearly what ASYNC_WAL means). Worse, old-time users probably associate
ASYNC_WAL and DEFERRED_FLUSH, an old HTableDescriptor config that was deprecated and replaced
by ASYNC_WAL. DEFERRED_FLUSH ran a background thread -- LogSyncer -- that on a configurable
interval, sent a sync down the write pipeline so any outstanding appends since last last interval
start get pushed out to the DN.  ASYNC_WAL doesn't do this (see below for history on how we
let go of the LogSyncer feature).
> Of note, we always sync meta edits. You can't turn this off. Also, given WALs are per
regionserver, if other regions on the RS are from tables that have sync set, these writes
will push out to the DN any appends done on tables that have DEFERRED/ASYNC_WAL set.
> To fix, we could do a few things:
>  * Simple and comprehensive would be always queuing a sync, even if ASYNC_WAL is set
but we let go of Handlers as soon as we write the memstore -- we don't wait on the sync to
complete as we do with the default setting of Durability=SYNC_WAL.
>  * Be like a 'real' database and add in a sync after N bytes of data have been appended
(configurable) or after M milliseconds have passed, which ever threshold happens first. The
size check would be easy. The sync-ever-M-millis would mean another thread.
>  * Doc what ASYNC_WAL means (and other durability options)
> Let me take a look and report back. Will file a bit of history on how we got here in
next comment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message