zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-3179) Add snapshot compression to reduce the disk IO
Date Wed, 10 Apr 2019 16:46:00 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814633#comment-16814633
] 

Hudson commented on ZOOKEEPER-3179:
-----------------------------------

SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #470 (See [https://builds.apache.org/job/ZooKeeper-trunk/470/])
ZOOKEEPER-3179: Add snapshot compression to reduce the disk IO (andor: rev 73cdd09051df33189351f7c42131724e3270eeac)
* (edit) ivy.xml
* (add) zookeeper-server/src/test/java/org/apache/zookeeper/server/persistence/SnapStreamTest.java
* (add) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/SnapStream.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/SnapshotFormatter.java
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/FileSnap.java
* (edit) build.xml
* (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/Util.java
* (edit) zookeeper-server/src/test/java/org/apache/zookeeper/server/ZooKeeperServerTest.java
* (edit) zookeeper-server/pom.xml


> Add snapshot compression to reduce the disk IO
> ----------------------------------------------
>
>                 Key: ZOOKEEPER-3179
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3179
>             Project: ZooKeeper
>          Issue Type: Improvement
>            Reporter: Fangmin Lv
>            Assignee: Yisong Yue
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.6.0
>
>          Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> When the snapshot becomes larger, the periodically snapshot after certain number of txns will
be more expensive. Which will in turn affect the maximum throughput we can support within
SLA, because of the disk contention between snapshot and txn when they're on the same drive.
>  
> With compression like zstd/snappy/gzip, the actual snapshot size could be much smaller,
the compress ratio depends on the actual data. It might make the recovery time (loading from
disk) faster in some cases, but will take longer sometimes because of the extra time used
to compress/decompress.
>  
> Based on the production traffic, the performance various with different compress method
as well, that's why we provided different implementations, we can select different compress
method for different use cases.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message