kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Otto <o...@wikimedia.org>
Subject Brokers changing mtime on data files during startup?
Date Wed, 25 May 2016 16:34:13 GMT
Hiya,

We’ve recently upgraded to 0.9.  In 0.8, when we restarted a broker, data
log file mtimes were not changed.  In 0.9, any data log file that was on
disk before the broker has it’s mtime modified to the time of the broker
restart.

This causes problems with log retention, as all the files then look like
they contain recent data to kafka.  We use the default log retention of 7
weeks, but if all the files are touched at the same time, this can cause us
to retain up to 2 weeks of log data, which can fill up our disks.

We saw this during our initial upgrade, but I had just thought it had
something to do with the change of inter.broker.protocol.version, and
assumed it wouldn’t happen again.  We just did our first broker restart
after the upgrade, and we are seeing this again.  We worked around this
during our upgrade by temporarily setting a high volume topic’s retention
very low, causing brokers to purge more recent data.  This allowed us to
avoid filling up our disks, but we shouldn’t have to do this every time we
bounce brokers.

Has anyone else noticed this?

-Ao

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message