mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joseph Wu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-6267) Mesos master should periodically compact the replicated log
Date Mon, 03 Oct 2016 16:21:20 GMT

    [ https://issues.apache.org/jira/browse/MESOS-6267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542777#comment-15542777
] 

Joseph Wu commented on MESOS-6267:
----------------------------------

[~igmorv] Is working on a design doc for this.  I think he'll share it on the dev list soon.

> Mesos master should periodically compact the replicated log
> -----------------------------------------------------------
>
>                 Key: MESOS-6267
>                 URL: https://issues.apache.org/jira/browse/MESOS-6267
>             Project: Mesos
>          Issue Type: Improvement
>          Components: replicated log
>            Reporter: David Robinson
>            Priority: Minor
>
> The mesos master does not periodically compact the replicated log's leveldb, it grows
without bound. Compaction only occurs on startup, therefore in large clusters (10k+ agents)
where the master is restarted infrequently compaction at startup can take 2+ minutes to complete.
At Twitter we've seen the DB grow to 170+ GB in less than a month and compaction take >
2 minutes. Compaction reduces storage usage from 160 GB to roughly 50 MB.
> {noformat}
> I0928 06:06:20.776908 19725 leveldb.cpp:174] Opened db in 763.082271ms
> I0928 06:08:41.636145 19725 leveldb.cpp:181] Compacted db in 2.34765004808333mins
> {noformat}
> The replicated log should not grow without bounds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message