mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Robinson (JIRA)" <>
Subject [jira] [Created] (MESOS-6267) Mesos master should periodically compact the replicated log
Date Wed, 28 Sep 2016 18:29:20 GMT
David Robinson created MESOS-6267:

             Summary: Mesos master should periodically compact the replicated log
                 Key: MESOS-6267
             Project: Mesos
          Issue Type: Improvement
          Components: replicated log
            Reporter: David Robinson
            Priority: Minor

The mesos master does not periodically compact the replicated log's leveldb, it grows without
bound. Compaction only occurs on startup, therefore in large clusters (10k+ agents) where
the master is restarted infrequently compaction at startup can take 2+ minutes to complete.
At Twitter we've seen the DB grow to 170+ GB in less than a month and compaction take >
2 minutes. Compaction reduces storage usage from 160 GB to roughly 50 MB.

I0928 06:06:20.776908 19725 leveldb.cpp:174] Opened db in 763.082271ms
I0928 06:08:41.636145 19725 leveldb.cpp:181] Compacted db in 2.34765004808333mins

The replicated log should not grow without bounds.

This message was sent by Atlassian JIRA

View raw message