hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL
Date Sat, 22 Oct 2016 01:37:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15596906#comment-15596906
] 

stack commented on HBASE-15709:
-------------------------------

Block size seems fine as limit. Block would run on till opportunity to roll so could be bigger
than 256M, right?

DFSClient#DFSOS has its own machinery. There is nought we can do to influence its operation.
Changing Q size or packet size doesn't matter. We are careful aggregating syncs and appends
on our side but then the DFSClient#DFSOS machinery does its own thing regardless (we need
to be in control of the packet sending). Given this, AsyncWAL for this reason and others is
the path forward.

Let us know how can help. Will look at Rams perf issue this w/e.

> Handle large edits for asynchronous WAL
> ---------------------------------------
>
>                 Key: HBASE-15709
>                 URL: https://issues.apache.org/jira/browse/HBASE-15709
>             Project: HBase
>          Issue Type: Sub-task
>          Components: io, wal
>            Reporter: Duo Zhang
>            Priority: Critical
>
> First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is larger than
PacketReceiver.MAX_PACKET_SIZE(16MB).
> Second, since we only allow one block here, we need to make sure we do not exceed the
block size after writing a large chunk of data.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message