kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephane Maarek <steph...@simplemachines.com.au>
Subject Speeding up Kafka Start cold startup time
Date Tue, 26 Sep 2017 05:01:05 GMT
Hi,

 

We have our Kafka Brokers in AWS backed by st1 EBS drives. These are optimised for throughput

On a kafka restart (warm restart), if some of the appropriate data is in the pagecache, everything
goes well and the broker boots in just a few seconds. (1000 partitions)

On a cold start (new EC2 instance which just got the volume attached), nothing is in the page
cache and the first boot can take up to 10 minutes. 

 

Question: what does the kafka broker do and read on start? I would assume index files and
so on, but not the actual segments, does it?

 

If so, would it be worth considering a KIP that would allow to optimise kafka files this way:
Kafka index files, or any small files, would be placed on a faster drive (say a gp2 SSD drive)
Kafka segment files, or any huge files, would be placed on a throughput optimised drive. 
 

I am not sure how the KIP would shape up, or the code change, but:
Is that a possible strategy to drastically improve the broker start-up time? 
Would that also improve shutdown time?
 

Thanks,

Stephane


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message