hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14352) Replication is terribly slow with WAL compression
Date Wed, 16 Sep 2015 20:09:49 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791052#comment-14791052
] 

Lars Hofhansl commented on HBASE-14352:
---------------------------------------

I took a look at the code some weeks back. The problem immediately jumps out... At the source
we constantly reset the read position into the current WAL. With compression it means we have
start from a point where the compression dictionary is written. That is very expensive.

We have to do that in order to be sure we'll see the edits in the current block being written.
So I don't see immediately a way out of it. Perhaps we simply tail until we reach the end
of a file. And that case we'll try one more time with a reset, and only declare the WAL done
when that is done.

> Replication is terribly slow with WAL compression
> -------------------------------------------------
>
>                 Key: HBASE-14352
>                 URL: https://issues.apache.org/jira/browse/HBASE-14352
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.13
>            Reporter: Abhishek Singh Chouhan
>         Attachments: age_of_last_shipped.png, size_of_log_queue.png
>
>
> For the same load, replication with WAL compression enabled is almost 6x slower than
with compression turned off. Age of last shipped operation is also correspondingly much higher
when compression is turned on. 
> By observing Size of log queue we can see that it is taking too much time for the queue
to clear up.
> Attaching corresponding graphs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message