storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <roshan_n...@yahoo.com>
Subject Re: Storm 2.0 and memory for worker
Date Sun, 11 Aug 2019 23:32:59 GMT
 Set  `storm.scheduler` to one of these classes deriving from IScheduler:
- org.apache.storm.scheduler.DefaultScheduler
- org.apache.storm.scheduler.EvenScheduler- org.apache.storm.scheduler.resource.ResourceAwareScheduler
-roshan

    On Saturday, August 10, 2019, 11:06:52 AM PDT, Michal Koziorowski <Michal.Koziorowski@tomtom.com>
wrote:  
 
 
Thanks for explanation. Do you know how to disable resource aware sheduler? I don't really
need it. I have only one topology with known constant number of workers, so I need very basic
sheduler.

Best regards,
Michal
From: Roshan Naik <roshan_naik@yahoo.com>
Sent: Saturday, 10 August 2019 07:45:41
To: user@storm.apache.org <user@storm.apache.org>
Subject: Re: Storm 2.0 and memory for worker   
I think the change in behavior has to do with the resource aware scheduler.  I have noticed
this setting impact the heap size. 
- topology.component.resources.onheap.memory.mb 

It (i think) allocates that much mem to each component instance running in any given worker.
So each worker's heap can be diff. 
If RAS is disabled, I suspect the behavior *might* revert to what you are used to 1.x. Perhaps
someone better familiar with RAS can comment.

-roshan



On Thursday, August 1, 2019, 01:23:53 AM PDT, Stig Rohde Døssing <stigdoessing@gmail.com>
wrote:





Here are a couple of links to the source that might help you.

The system property is coming from https://github.com/apache/storm/blob/eaed3cbf3610bbefb9466f2c40db1e0e2a20ca9c/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L574

It is being calculated by https://github.com/apache/storm/blob/eaed3cbf3610bbefb9466f2c40db1e0e2a20ca9c/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L805

See also https://github.com/apache/storm/blob/eaed3cbf3610bbefb9466f2c40db1e0e2a20ca9c/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L579

Den ons. 31. jul. 2019 kl. 18.41 skrev Michal Koziorowski <Michal.Koziorowski@tomtom.com>:
>  
>  
>  Hi,
> 
>  
> 
> 
>  In storm 1.1 I could set Xmx for a worker by setting worker.heap.memory.mb in storm.yaml.
> 
>  It seems ignored in my upgraded Storm 2.0 configuration and Xmx value was taken from -Dworker.memory_limit_mb=5888
which I didn't set.
> 
>  How to make setting from storm 1.1 working in 2.0?
> 
>  
> 
> 
>  I have following properties in storm.yaml:
> 
>  worker.heap.memory.mb: 16000
> 
> 
>  worker.childopts: "-Xmx%HEAP-MEM%m -Xlog:gc*,gc+age=trace,safepoint:file=artifacts/gc.log:utctime,pid,tags:filecount=32,filesize=10m
-Djava.net.preferIPv4Stack=true"
> 
> 
>  
> 
> 
>  Previously 16000 was used as %HEAP-MEM%
> 
>  
> 
> 
> 
  
Mime
View raw message