hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
Date Tue, 25 Oct 2016 01:47:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15603867#comment-15603867

Duo Zhang commented on HBASE-16890:

The DFSOutputStream works like this

1. Copy data into the buf in FSOutputSummer
2. If the size exceeds limit(checksumChunkSize * BUFFER_NUM_CHUNKS), then compute the checksum
and call writeChunk method in DFSOutputStream
3. Serialize the chunk data into currentPacket field in DFSOutputStream.
4. When flushing, compute the checksum of the remaining data in buf if any, call writeChunk,
and then write the currentPacket out.

And for FanOutOneBlockAsyncDFSOutput, the processing is straight forward. Buffer all data
in buf, and do all other things such as compute checksum, construct packet when flushing.

I do not think this will cause much difference unless we write very large data at the same
time that we maybe blocked in the JNI call for a long time which has some bad impact on gc.

> Analyze the performance of AsyncWAL and fix the same
> ----------------------------------------------------
>                 Key: HBASE-16890
>                 URL: https://issues.apache.org/jira/browse/HBASE-16890
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>    Affects Versions: 2.0.0
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: ramkrishna.s.vasudevan
>             Fix For: 2.0.0
>         Attachments: contention.png, contention_defaultWAL.png
> Tests reveal that AsyncWAL under load in single node cluster performs slower than the
Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.

This message was sent by Atlassian JIRA

View raw message