apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vlad Rozov <v.ro...@datatorrent.com>
Subject Re: Question about tweaking memory controls
Date Sat, 26 Sep 2015 23:25:37 GMT
In the benchmark test THREAD_LOCAL outperforms CONTAINER_LOCAL by an 
order of magnitude and both operators compete for CPU. I'll take a 
closer look why.

Thank you,

Vlad

On 9/26/15 14:52, Thomas Weise wrote:
> THREAD_LOCAL - operators share thread
> CONTAINER_LOCAL - each operator has its own thread
>
> So as long as operators utilize the CPU sufficiently (compete), the latter
> will perform better.
>
> There will be cases where a single thread can accommodate multiple
> operators. For example, a socket reader (mostly waiting for IO) and a
> decompress (CPU hungry) can share a thread.
>
> But to get back to the original question, stream locality does generally
> not reduce the total memory requirement. If you add multiple operators into
> one container, that container will also require more memory and that's how
> the container size is calculated in the physical plan. You may get some
> extra mileage when multiple operators share the same heap but the need to
> identify the memory requirement per operator does not go away.
>
> Thomas
>
>
> On Sat, Sep 26, 2015 at 12:41 PM, Munagala Ramanath <ram@datatorrent.com>
> wrote:
>
>> Would CONTAINER_LOCAL achieve the same thing and perform a little better on
>> a multi-core box ?
>>
>> Ram
>>
>> On Sat, Sep 26, 2015 at 12:18 PM, Sandeep Deshmukh <
>> sandeep@datatorrent.com>
>> wrote:
>>
>>> Yes, with this approach only two containers are required: one for stram
>> and
>>> another for all operators. You can easily fit around 10 operators in less
>>> than 1GB.
>>> On 27 Sep 2015 00:32, "Timothy Farkas" <tim@datatorrent.com> wrote:
>>>
>>>> Hi Ram,
>>>>
>>>> You could make all the operators thread local. This cuts down on the
>>>> overhead of separate containers and maximizes the memory available to
>>> each
>>>> operator.
>>>>
>>>> Tim
>>>>
>>>> On Sat, Sep 26, 2015 at 10:07 AM, Munagala Ramanath <
>> ram@datatorrent.com
>>>> wrote:
>>>>
>>>>>   Hi,
>>>>>
>>>>> I was running into memory issues when deploying my  app on the
>> sandbox
>>>>> where all the operators were stuck forever in the PENDING state
>> because
>>>>> they were being continually aborted and restarted because of the
>>> limited
>>>>> memory on the sandbox. After some experimentation, I found that the
>>>>> following config values seem to work:
>>>>> ------------------------------------------
>>>>> <
>> https://datatorrent.slack.com/archives/engineering/p1443263607000010>
>>>>>
>>>>>
>>>>> *<property>    <name>dt.attr.MASTER_MEMORY_MB</name>
>>>> <value>500</value>
>>>>>   </property>  <property>    <name>dt.application.​.operator.*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *​.attr.MEMORY_MB</name>    <value>200</value>  </property>
>> <property>
>>>>>
>> <name>dt.application.TopNWordsWithQueries.operator.fileWordCount.attr.MEMORY_MB</name>
>>>>>     <value>512</value>  </property>*
>>>>> ------------------------------------------------
>>>>> Are these reasonable values ? Is there a more systematic way of
>> coming
>>> up
>>>>> with these values than trial-and-error ? Most of my operators -- with
>>> the
>>>>> exception of fileWordCount -- need very little memory; is there a way
>>> to
>>>>> cut all values down to the bare minimum and maximize available memory
>>> for
>>>>> this one operator ?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Ram
>>>>>


Mime
View raw message