apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siyuan Hua <siy...@datatorrent.com>
Subject Re: Question about tweaking memory controls
Date Sat, 26 Sep 2015 19:40:53 GMT
Instead of tweak memory settings I think you can tweak memory setting for yarn.  In this way,
you can launch applications with more operators with less risk of outofmemory error and also
show different localities 

Sent from my iPhone

> On Sep 26, 2015, at 12:18, Sandeep Deshmukh <sandeep@datatorrent.com> wrote:
> 
> Yes, with this approach only two containers are required: one for stram and
> another for all operators. You can easily fit around 10 operators in less
> than 1GB.
>> On 27 Sep 2015 00:32, "Timothy Farkas" <tim@datatorrent.com> wrote:
>> 
>> Hi Ram,
>> 
>> You could make all the operators thread local. This cuts down on the
>> overhead of separate containers and maximizes the memory available to each
>> operator.
>> 
>> Tim
>> 
>> On Sat, Sep 26, 2015 at 10:07 AM, Munagala Ramanath <ram@datatorrent.com>
>> wrote:
>> 
>>> Hi,
>>> 
>>> I was running into memory issues when deploying my  app on the sandbox
>>> where all the operators were stuck forever in the PENDING state because
>>> they were being continually aborted and restarted because of the limited
>>> memory on the sandbox. After some experimentation, I found that the
>>> following config values seem to work:
>>> ------------------------------------------
>>> <https://datatorrent.slack.com/archives/engineering/p1443263607000010>
>>> 
>>> 
>>> 
>>> *<property>    <name>dt.attr.MASTER_MEMORY_MB</name>
>> <value>500</value>
>>> </property>  <property>    <name>dt.application.​.operator.*
>>> 
>>> 
>>> 
>>> 
>>> 
>>> *​.attr.MEMORY_MB</name>    <value>200</value>  </property>
 <property>
>> <name>dt.application.TopNWordsWithQueries.operator.fileWordCount.attr.MEMORY_MB</name>
>>>   <value>512</value>  </property>*
>>> ------------------------------------------------
>>> Are these reasonable values ? Is there a more systematic way of coming up
>>> with these values than trial-and-error ? Most of my operators -- with the
>>> exception of fileWordCount -- need very little memory; is there a way to
>>> cut all values down to the bare minimum and maximize available memory for
>>> this one operator ?
>>> 
>>> Thanks.
>>> 
>>> Ram
>> 

Mime
View raw message