kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Memory consumption of Kafka-examples Kafka-streams around 1.5 GB
Date Wed, 15 Jun 2016 22:25:26 GMT
Hello Phillippe,

I used to run the "SimpleBenchmark" on my laptop with 4GB also, and it
usually used close to, but less than 1GB.

https://www.codatlas.com/github.com/apache/kafka/HEAD/streams/src/test/java/org/apache/kafka/streams/perf/SimpleBenchmark.java

Note that I need to bootstrap a real ZK instance and a Kafka instance in
order to run that benchmark, and I think those two instances are actually
taking the major memory usage than Kafka Streams instance itself.

There may be some extra memory overhead from maven framework but I would be
surprised if that is taking large amount.

Which two test cases specifically are causing OOMs on your laptop?

Guozhang


On Tue, Jun 14, 2016 at 4:50 PM, Philippe Derome <phderome@gmail.com> wrote:

> I am running "mvn test" as per tip from
> https://github.com/confluentinc/examples/tree/master/kafka-streams
> README.MD.
> This uses embedded Kafka components from test alone (no ZK, Kafka,
> schema-registry running).
>
> I monitor on OSX El Capitan (10.11.5) memory usage and it grows on Java
> processes from nothing to about 1.3GB when it fails to execute last 2 tests
> with Java out of memory exceptions. Selecting 2 tests to avoid makes the
> test pass but with my 4GB system, I cannot pass them all.
>
> Is that relatively large memory consumption to be expected on these test
> cases?
>
> I'd like to run stand-alone from jar and I'll be able to do so by excluding
> test cases.
>
> Fyi, on mailing list I see only 1 issue related to Streams and memory if
> that's any relevant (KAFKA-3738).
>
> Phil
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message