hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ramkrishna.s.vasudevan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15536) Make AsyncFSWAL as our default WAL
Date Tue, 18 Oct 2016 11:55:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585269#comment-15585269

ramkrishna.s.vasudevan commented on HBASE-15536:

As said, we are doing some experiments to check how we can run with AsyncWAL when we have
the write path in offheap structures. This JIRA creates an outputStream which allows to work
with ByteBuffers. 
But overall we see a significant degrade just in using the current trunk with ASYNC_WAL and
With PE tool with 10 cols and 50 threads
With Default WAL

With Async WAL

Since this is a single node system the load on the DN and NN is heavy and so the msg
bq.Slow sync cost: 166 ms, current pipeline: [DatanodeInfoWithStorage[,DS-bc84f329-2043-4891-b878-f368c774b246,DISK]]

The thread model between the default case and the ASYNC WALbasically tries to be the same.
We have some contentions with the ArrayDeque that is in AsyncWAL now. 
But the major difference I could see is that in the default WAL case the checksum is being
done in the append() flow because append() adds the data to the FSDataOS.
In the AsyncWAL we add the data to the new BAOS but it is the sync call that actually does
the check summing. 

Interesting thing is that when we run the WALPE tool with 50 threads with key size 50 : val
: 200 , AsyncWAL is performing better in terms of ops/sec.
Default WAL
Summary: threads=50, iterations=1000000, syncInterval=0 took 1475.368s 33889.848ops/s

Summary: threads=50, iterations=1000000, syncInterval=0 took 1368.501s 36536.328ops/s

> Make AsyncFSWAL as our default WAL
> ----------------------------------
>                 Key: HBASE-15536
>                 URL: https://issues.apache.org/jira/browse/HBASE-15536
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>    Affects Versions: 2.0.0
>            Reporter: Duo Zhang
>            Assignee: Duo Zhang
>            Priority: Critical
>             Fix For: 2.0.0
>         Attachments: HBASE-15536-v1.patch, HBASE-15536-v2.patch, HBASE-15536-v3.patch,
HBASE-15536-v4.patch, HBASE-15536-v5.patch, HBASE-15536.patch
> As it should be predicated on passing basic cluster ITBLL

This message was sent by Atlassian JIRA

View raw message