apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Default App Master Container Memory Size
Date Mon, 28 Dec 2015 19:54:13 GMT
Gaurav,

If increasing the default is not an option, could you please look into the
issue? Our users are getting burned by this and it's difficult to diagnose
unless you've seen it before.

https://issues.apache.org/jira/browse/APEXCORE-296

Thanks,
Tim

On Wed, Dec 23, 2015 at 10:46 PM, Aniruddha Thombare <
aniruddha@datatorrent.com> wrote:

> Hi,
>
> In my demo setups I have observed this multiple times that app master dies
> on 1024MB memory.
>
> So, I have set default app master to be at 2GB and since then haven't
> observed the issue.
>
> But, I would say that we should check why we need that much memory for app
> master as Gaurav suggested.
>
> Platform should use minimal possible memory and keep resources free for
> actual applications.
>
> Thanks,
>
> Aniruddha
>
> Sent from handheld device...
> On Dec 24, 2015 11:23 AM, "Chinmay Kolhatkar" <chinmay@datatorrent.com>
> wrote:
>
> > I agree with Gaurav.
> > We should try to identify what is causing current memory limit not to
> work.
> >
> > If there is a need for a stop gap solution, we could temporarily make it
> > higher. But I believe this is not a solution unless we answer the "why"
> > question.
> >
> >
> > ~ Chinmay.
> >
> > On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <gaurav@datatorrent.com>
> > wrote:
> >
> > > Tim,
> > >
> > > This setting used to work fine earlier. What did we change/add in App
> > > master that is causing this behavior. We should know the reason before
> > > bumping the memory. May be there is some other issue.
> > >
> > > Thanks
> > > - Gaurav
> > >
> > > > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <
> sandeep@datatorrent.com
> > >
> > > wrote:
> > > >
> > > > I am ok with 1280MB.
> > > > On 24 Dec 2015 06:34, "Timothy Farkas" <tim@datatorrent.com> wrote:
> > > >
> > > >> If there are no objections I will open a pull request shortly to
> bump
> > up
> > > >> the default App Master Memory to 1280 MB. I recently got burned by
> > this
> > > >> default AGAIN when the App Master randomly died in the middle of
> demo.
> > > The
> > > >> demo application was small and only had 6 physical operators.
> > > >>
> > > >> Thanks,
> > > >> Tim
> > > >>
> > > >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <
> > ram@datatorrent.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Sorry, meant to say 4-6GB, not MB.
> > > >>>
> > > >>> Ram
> > > >>>
> > > >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <
> > > ram@datatorrent.com>
> > > >>> wrote:
> > > >>>
> > > >>>> I'm OK with increasing it to 1280MB.
> > > >>>>
> > > >>>> I agree with Sandeep about new people trying out Apex,
> > > >>>> often on limited environments like VMs limited to 4-6MB. With
a
> > > default
> > > >>> of
> > > >>>> 0.5GB per Buffer Server
> > > >>>> and 1GB per operator, things start to fail when the user has
more
> > than
> > > >> a
> > > >>>> handful of operators.
> > > >>>>
> > > >>>> I'm guessing it's not just the number of the operators that
puts
> > > memory
> > > >>>> pressure on master memory
> > > >>>> but the size of the jars that get pulled in by all the operator
> > > classes
> > > >>>> and their dependencies. If this is
> > > >>>> true, one could cause failure even with a single, sufficiently
> > complex
> > > >>>> operator.
> > > >>>>
> > > >>>> Ram
> > > >>>>
> > > >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <
> > tim@datatorrent.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi Sandeep,
> > > >>>>>
> > > >>>>> I have seen the issue even with small apps like Twitter
Demo. It
> > > >> happens
> > > >>>>> less frequently in very small apps but the issue is still
there.
> It
> > > >>>>> happens
> > > >>>>> frequently in medium size apps (10 containers). Could
we increase
> > the
> > > >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we
can get
> > greater
> > > >>>>> stability with the defaults settings without allocating
too much
> > > >>>>> additional
> > > >>>>> memory?
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Tim
> > > >>>>>
> > > >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> > > >>>>> sandeep@datatorrent.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> With large number of containers, the problem becomes
severe and
> I
> > > >> too
> > > >>>>> have
> > > >>>>>> faced it.
> > > >>>>>>
> > > >>>>>> Not sure if default should be increased as that should
work in
> > most
> > > >> of
> > > >>>>> the
> > > >>>>>> cases and even when new people are trying out Apex
for the first
> > > >> time.
> > > >>>>>> They may not have enough memory.
> > > >>>>>>
> > > >>>>>> Regards
> > > >>>>>> Sandeep
> > > >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <
> > Ilya.Ganelin@capitalone.com>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Tim - great point, we reduce the memory consumption
for
> operators
> > > >>> and
> > > >>>>> do
> > > >>>>>>> actually increase it for the app master, especially
when we
> have
> > > >>> more
> > > >>>>>>> operators. I misread your original email.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Thank you,
> > > >>>>>>> Ilya Ganelin
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> -----Original Message-----
> > > >>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > > >>> tim@datatorrent.com
> > > >>>>>> ]
> > > >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern
Standard
> Time
> > > >>>>>>> To: dev@apex.incubator.apache.org
> > > >>>>>>> Subject: Re: Default App Master Container Memory
Size
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Hi Ilya,
> > > >>>>>>>
> > > >>>>>>> Dou you reduce the memory to 256 mb for operators
only or for
> the
> > > >>> App
> > > >>>>>>> Master as well?
> > > >>>>>>>
> > > >>>>>>> Thanks,
> > > >>>>>>> Tim
> > > >>>>>>>
> > > >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya
<
> > > >>>>>>> Ilya.Ganelin@capitalone.com>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Tim - in our applications we've actually tried
to go with
> > > >> smaller
> > > >>>>>>>> containers, as small as 256 MB, simply because
many operators
> > > >>> don't
> > > >>>>>>> demand
> > > >>>>>>>> any memory and this allows us to run more
partitions.
> > > >>>>>>>>
> > > >>>>>>>> We haven't noticed performance degradation.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Thank you,
> > > >>>>>>>> Ilya Ganelin
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> -----Original Message-----
> > > >>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > > >>>>> tim@datatorrent.com>]
> > > >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM
Eastern Standard
> Time
> > > >>>>>>>> To: dev@apex.incubator.apache.org
> > > >>>>>>>> Subject: Default App Master Container Memory
Size
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Hi All,
> > > >>>>>>>>
> > > >>>>>>>> The default value for MASTER_MEMORY_MB is
1024. I've observed
> in
> > > >>>>>> multiple
> > > >>>>>>>> demo apps and in the production apps of some
of our users that
> > > >>> 1024
> > > >>>>> MB
> > > >>>>>> is
> > > >>>>>>>> not enough memory, and the app master dies.
This was observed
> on
> > > >>>>> small
> > > >>>>>>>> applications with 5 - 20 containers. I think
the default app
> > > >>> master
> > > >>>>>>> memory
> > > >>>>>>>> size should be 2048.
> > > >>>>>>>>
> > > >>>>>>>> Any feedback?
> > > >>>>>>>>
> > > >>>>>>>> Thanks,
> > > >>>>>>>> Tim
> > > >>>>>>>> ________________________________________________________
> > > >>>>>>>>
> > > >>>>>>>> The information contained in this e-mail is
confidential
> and/or
> > > >>>>>>>> proprietary to Capital One and/or its affiliates
and may only
> be
> > > >>>>> used
> > > >>>>>>>> solely in performance of work or services
for Capital One. The
> > > >>>>>>> information
> > > >>>>>>>> transmitted herewith is intended only for
use by the
> individual
> > > >> or
> > > >>>>>> entity
> > > >>>>>>>> to which it is addressed. If the reader of
this message is not
> > > >> the
> > > >>>>>>> intended
> > > >>>>>>>> recipient, you are hereby notified that any
review,
> > > >>> retransmission,
> > > >>>>>>>> dissemination, distribution, copying or other
use of, or
> taking
> > > >> of
> > > >>>>> any
> > > >>>>>>>> action in reliance upon this information is
strictly
> prohibited.
> > > >>> If
> > > >>>>> you
> > > >>>>>>>> have received this communication in error,
please contact the
> > > >>> sender
> > > >>>>>> and
> > > >>>>>>>> delete the material from your computer.
> > > >>>>>>>>
> > > >>>>>>> ________________________________________________________
> > > >>>>>>>
> > > >>>>>>> The information contained in this e-mail is confidential
and/or
> > > >>>>>>> proprietary to Capital One and/or its affiliates
and may only
> be
> > > >>> used
> > > >>>>>>> solely in performance of work or services for
Capital One. The
> > > >>>>>> information
> > > >>>>>>> transmitted herewith is intended only for use
by the individual
> > or
> > > >>>>> entity
> > > >>>>>>> to which it is addressed. If the reader of this
message is not
> > the
> > > >>>>>> intended
> > > >>>>>>> recipient, you are hereby notified that any review,
> > > >> retransmission,
> > > >>>>>>> dissemination, distribution, copying or other
use of, or taking
> > of
> > > >>> any
> > > >>>>>>> action in reliance upon this information is strictly
> prohibited.
> > > >> If
> > > >>>>> you
> > > >>>>>>> have received this communication in error, please
contact the
> > > >> sender
> > > >>>>> and
> > > >>>>>>> delete the material from your computer.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message