hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duo Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL
Date Sat, 22 Oct 2016 00:19:58 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596763#comment-15596763
] 

Duo Zhang commented on HBASE-15709:
-----------------------------------

I can add the support of entries larger than 16M, as you said, split it into several chunks.
But here comes another limit, the block size. Our fan out output is 'one block'...

So there will always be a size limit for AsyncFSWAL. 16M or 256M. And the two WALs will break
our serial replication feature. So if we decide to use AsyncFSWAL then we must reject the
large edits...

If you guys think 16M is too small and 256M is OK then I could try to add chunking support
for AsyncFSWAL.

Thanks.

> Handle large edits for asynchronous WAL
> ---------------------------------------
>
>                 Key: HBASE-15709
>                 URL: https://issues.apache.org/jira/browse/HBASE-15709
>             Project: HBase
>          Issue Type: Sub-task
>          Components: io, wal
>            Reporter: Duo Zhang
>            Priority: Critical
>
> First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is larger than
PacketReceiver.MAX_PACKET_SIZE(16MB).
> Second, since we only allow one block here, we need to make sure we do not exceed the
block size after writing a large chunk of data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message