kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Damian Guy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-4484) Set more conservative default values on RocksDB for memory usage
Date Thu, 09 Feb 2017 14:42:41 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Damian Guy updated KAFKA-4484:
    Status: Patch Available  (was: In Progress)

> Set more conservative default values on RocksDB for memory usage
> ----------------------------------------------------------------
>                 Key: KAFKA-4484
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4484
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Guozhang Wang
>            Assignee: Damian Guy
>              Labels: performance
> Quoting from email thread:
> {code}
> The block cache size defaults to a whopping 100Mb per store, and that gets
> expensive
> fast. I reduced it to a few megabytes. My data size is so big that I doubt
> it is very effective
> anyway. Now it seems more stable.
> I'd say that a smaller default makes sense, especially because the failure
> case is
> so opaque (running all tests just fine but with a serious dataset it dies
> slowly)
> {code}
> {code}
> Before we have the a single-knob memory management feature, I'd like to propose reducing
the Streams' default config values for RocksDB caching and memory block size. For example,
I remember Henry has done some fine tuning on the RocksDB config for his use case:
> https://github.com/HenryCaiHaiying/kafka/commit/b297f7c585f5a883ee068277e5f0f1224c347bd4
> https://github.com/HenryCaiHaiying/kafka/commit/eed1726d16e528d813755a6e66b49d0bf14e8803
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
> We could check if some of those changes are appropriate in general and if yes change
the default settings accordingly.
> {code}

This message was sent by Atlassian JIRA

View raw message