storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Koziorowski <Michal.Koziorow...@tomtom.com>
Subject Re: Storm 2.0 and memory for worker
Date Sat, 10 Aug 2019 18:06:33 GMT

Thanks for explanation. Do you know how to disable resource aware sheduler? I don't really
need it. I have only one topology with known constant number of workers, so I need very basic
sheduler.

Best regards,
Michal
________________________________
From: Roshan Naik <roshan_naik@yahoo.com>
Sent: Saturday, 10 August 2019 07:45:41
To: user@storm.apache.org <user@storm.apache.org>
Subject: Re: Storm 2.0 and memory for worker


I think the change in behavior has to do with the resource aware scheduler.  I have noticed
this setting impact the heap size.
- topology.component.resources.onheap.memory.mb

It (i think) allocates that much mem to each component instance running in any given worker.
So each worker's heap can be diff.
If RAS is disabled, I suspect the behavior *might* revert to what you are used to 1.x. Perhaps
someone better familiar with RAS can comment.

-roshan



On Thursday, August 1, 2019, 01:23:53 AM PDT, Stig Rohde Døssing <stigdoessing@gmail.com>
wrote:





Here are a couple of links to the source that might help you.

The system property is coming from https://github.com/apache/storm/blob/eaed3cbf3610bbefb9466f2c40db1e0e2a20ca9c/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L574

It is being calculated by https://github.com/apache/storm/blob/eaed3cbf3610bbefb9466f2c40db1e0e2a20ca9c/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L805

See also https://github.com/apache/storm/blob/eaed3cbf3610bbefb9466f2c40db1e0e2a20ca9c/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L579

Den ons. 31. jul. 2019 kl. 18.41 skrev Michal Koziorowski <Michal.Koziorowski@tomtom.com>:
>
>
>  Hi,
>
>
>
>
>  In storm 1.1 I could set Xmx for a worker by setting worker.heap.memory.mb in storm.yaml.
>
>  It seems ignored in my upgraded Storm 2.0 configuration and Xmx value was taken from
-Dworker.memory_limit_mb=5888 which I didn't set.
>
>  How to make setting from storm 1.1 working in 2.0?
>
>
>
>
>  I have following properties in storm.yaml:
>
>  worker.heap.memory.mb: 16000
>
>
>  worker.childopts: "-Xmx%HEAP-MEM%m -Xlog:gc*,gc+age=trace,safepoint:file=artifacts/gc.log:utctime,pid,tags:filecount=32,filesize=10m
-Djava.net.preferIPv4Stack=true"
>
>
>
>
>
>  Previously 16000 was used as %HEAP-MEM%
>
>
>
>
>

Mime
View raw message