apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Question about tweaking memory controls
Date Sat, 26 Sep 2015 19:41:03 GMT
Would CONTAINER_LOCAL achieve the same thing and perform a little better on
a multi-core box ?

Ram

On Sat, Sep 26, 2015 at 12:18 PM, Sandeep Deshmukh <sandeep@datatorrent.com>
wrote:

> Yes, with this approach only two containers are required: one for stram and
> another for all operators. You can easily fit around 10 operators in less
> than 1GB.
> On 27 Sep 2015 00:32, "Timothy Farkas" <tim@datatorrent.com> wrote:
>
> > Hi Ram,
> >
> > You could make all the operators thread local. This cuts down on the
> > overhead of separate containers and maximizes the memory available to
> each
> > operator.
> >
> > Tim
> >
> > On Sat, Sep 26, 2015 at 10:07 AM, Munagala Ramanath <ram@datatorrent.com
> >
> > wrote:
> >
> > >  Hi,
> > >
> > > I was running into memory issues when deploying my  app on the sandbox
> > > where all the operators were stuck forever in the PENDING state because
> > > they were being continually aborted and restarted because of the
> limited
> > > memory on the sandbox. After some experimentation, I found that the
> > > following config values seem to work:
> > > ------------------------------------------
> > > <https://datatorrent.slack.com/archives/engineering/p1443263607000010>
> > >
> > >
> > >
> > > *<property>    <name>dt.attr.MASTER_MEMORY_MB</name>
> > <value>500</value>
> > >  </property>  <property>    <name>dt.application.​.operator.*
> > >
> > >
> > >
> > >
> > >
> > > *​.attr.MEMORY_MB</name>    <value>200</value>  </property>
 <property>
> > >
> > >
> >
> <name>dt.application.TopNWordsWithQueries.operator.fileWordCount.attr.MEMORY_MB</name>
> > >    <value>512</value>  </property>*
> > > ------------------------------------------------
> > > Are these reasonable values ? Is there a more systematic way of coming
> up
> > > with these values than trial-and-error ? Most of my operators -- with
> the
> > > exception of fileWordCount -- need very little memory; is there a way
> to
> > > cut all values down to the bare minimum and maximize available memory
> for
> > > this one operator ?
> > >
> > > Thanks.
> > >
> > > Ram
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message