apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniruddha Thombare <anirud...@datatorrent.com>
Subject Re: Default App Master Container Memory Size
Date Thu, 24 Dec 2015 06:46:56 GMT
Hi,

In my demo setups I have observed this multiple times that app master dies
on 1024MB memory.

So, I have set default app master to be at 2GB and since then haven't
observed the issue.

But, I would say that we should check why we need that much memory for app
master as Gaurav suggested.

Platform should use minimal possible memory and keep resources free for
actual applications.

Thanks,

Aniruddha

Sent from handheld device...
On Dec 24, 2015 11:23 AM, "Chinmay Kolhatkar" <chinmay@datatorrent.com>
wrote:

> I agree with Gaurav.
> We should try to identify what is causing current memory limit not to work.
>
> If there is a need for a stop gap solution, we could temporarily make it
> higher. But I believe this is not a solution unless we answer the "why"
> question.
>
>
> ~ Chinmay.
>
> On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <gaurav@datatorrent.com>
> wrote:
>
> > Tim,
> >
> > This setting used to work fine earlier. What did we change/add in App
> > master that is causing this behavior. We should know the reason before
> > bumping the memory. May be there is some other issue.
> >
> > Thanks
> > - Gaurav
> >
> > > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <sandeep@datatorrent.com
> >
> > wrote:
> > >
> > > I am ok with 1280MB.
> > > On 24 Dec 2015 06:34, "Timothy Farkas" <tim@datatorrent.com> wrote:
> > >
> > >> If there are no objections I will open a pull request shortly to bump
> up
> > >> the default App Master Memory to 1280 MB. I recently got burned by
> this
> > >> default AGAIN when the App Master randomly died in the middle of demo.
> > The
> > >> demo application was small and only had 6 physical operators.
> > >>
> > >> Thanks,
> > >> Tim
> > >>
> > >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <
> ram@datatorrent.com
> > >
> > >> wrote:
> > >>
> > >>> Sorry, meant to say 4-6GB, not MB.
> > >>>
> > >>> Ram
> > >>>
> > >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <
> > ram@datatorrent.com>
> > >>> wrote:
> > >>>
> > >>>> I'm OK with increasing it to 1280MB.
> > >>>>
> > >>>> I agree with Sandeep about new people trying out Apex,
> > >>>> often on limited environments like VMs limited to 4-6MB. With a
> > default
> > >>> of
> > >>>> 0.5GB per Buffer Server
> > >>>> and 1GB per operator, things start to fail when the user has more
> than
> > >> a
> > >>>> handful of operators.
> > >>>>
> > >>>> I'm guessing it's not just the number of the operators that puts
> > memory
> > >>>> pressure on master memory
> > >>>> but the size of the jars that get pulled in by all the operator
> > classes
> > >>>> and their dependencies. If this is
> > >>>> true, one could cause failure even with a single, sufficiently
> complex
> > >>>> operator.
> > >>>>
> > >>>> Ram
> > >>>>
> > >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <
> tim@datatorrent.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Sandeep,
> > >>>>>
> > >>>>> I have seen the issue even with small apps like Twitter Demo.
It
> > >> happens
> > >>>>> less frequently in very small apps but the issue is still there.
It
> > >>>>> happens
> > >>>>> frequently in medium size apps (10 containers). Could we increase
> the
> > >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get
> greater
> > >>>>> stability with the defaults settings without allocating too
much
> > >>>>> additional
> > >>>>> memory?
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Tim
> > >>>>>
> > >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> > >>>>> sandeep@datatorrent.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> With large number of containers, the problem becomes severe
and I
> > >> too
> > >>>>> have
> > >>>>>> faced it.
> > >>>>>>
> > >>>>>> Not sure if default should be increased as that should
work in
> most
> > >> of
> > >>>>> the
> > >>>>>> cases and even when new people are trying out Apex for
the first
> > >> time.
> > >>>>>> They may not have enough memory.
> > >>>>>>
> > >>>>>> Regards
> > >>>>>> Sandeep
> > >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <
> Ilya.Ganelin@capitalone.com>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Tim - great point, we reduce the memory consumption
for operators
> > >>> and
> > >>>>> do
> > >>>>>>> actually increase it for the app master, especially
when we have
> > >>> more
> > >>>>>>> operators. I misread your original email.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Thank you,
> > >>>>>>> Ilya Ganelin
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> -----Original Message-----
> > >>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > >>> tim@datatorrent.com
> > >>>>>> ]
> > >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern
Standard Time
> > >>>>>>> To: dev@apex.incubator.apache.org
> > >>>>>>> Subject: Re: Default App Master Container Memory Size
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Hi Ilya,
> > >>>>>>>
> > >>>>>>> Dou you reduce the memory to 256 mb for operators only
or for the
> > >>> App
> > >>>>>>> Master as well?
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Tim
> > >>>>>>>
> > >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> > >>>>>>> Ilya.Ganelin@capitalone.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Tim - in our applications we've actually tried
to go with
> > >> smaller
> > >>>>>>>> containers, as small as 256 MB, simply because
many operators
> > >>> don't
> > >>>>>>> demand
> > >>>>>>>> any memory and this allows us to run more partitions.
> > >>>>>>>>
> > >>>>>>>> We haven't noticed performance degradation.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Thank you,
> > >>>>>>>> Ilya Ganelin
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> -----Original Message-----
> > >>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> > >>>>> tim@datatorrent.com>]
> > >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern
Standard Time
> > >>>>>>>> To: dev@apex.incubator.apache.org
> > >>>>>>>> Subject: Default App Master Container Memory Size
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Hi All,
> > >>>>>>>>
> > >>>>>>>> The default value for MASTER_MEMORY_MB is 1024.
I've observed in
> > >>>>>> multiple
> > >>>>>>>> demo apps and in the production apps of some of
our users that
> > >>> 1024
> > >>>>> MB
> > >>>>>> is
> > >>>>>>>> not enough memory, and the app master dies. This
was observed on
> > >>>>> small
> > >>>>>>>> applications with 5 - 20 containers. I think the
default app
> > >>> master
> > >>>>>>> memory
> > >>>>>>>> size should be 2048.
> > >>>>>>>>
> > >>>>>>>> Any feedback?
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Tim
> > >>>>>>>> ________________________________________________________
> > >>>>>>>>
> > >>>>>>>> The information contained in this e-mail is confidential
and/or
> > >>>>>>>> proprietary to Capital One and/or its affiliates
and may only be
> > >>>>> used
> > >>>>>>>> solely in performance of work or services for Capital
One. The
> > >>>>>>> information
> > >>>>>>>> transmitted herewith is intended only for use by
the individual
> > >> or
> > >>>>>> entity
> > >>>>>>>> to which it is addressed. If the reader of this
message is not
> > >> the
> > >>>>>>> intended
> > >>>>>>>> recipient, you are hereby notified that any review,
> > >>> retransmission,
> > >>>>>>>> dissemination, distribution, copying or other use
of, or taking
> > >> of
> > >>>>> any
> > >>>>>>>> action in reliance upon this information is strictly
prohibited.
> > >>> If
> > >>>>> you
> > >>>>>>>> have received this communication in error, please
contact the
> > >>> sender
> > >>>>>> and
> > >>>>>>>> delete the material from your computer.
> > >>>>>>>>
> > >>>>>>> ________________________________________________________
> > >>>>>>>
> > >>>>>>> The information contained in this e-mail is confidential
and/or
> > >>>>>>> proprietary to Capital One and/or its affiliates and
may only be
> > >>> used
> > >>>>>>> solely in performance of work or services for Capital
One. The
> > >>>>>> information
> > >>>>>>> transmitted herewith is intended only for use by the
individual
> or
> > >>>>> entity
> > >>>>>>> to which it is addressed. If the reader of this message
is not
> the
> > >>>>>> intended
> > >>>>>>> recipient, you are hereby notified that any review,
> > >> retransmission,
> > >>>>>>> dissemination, distribution, copying or other use of,
or taking
> of
> > >>> any
> > >>>>>>> action in reliance upon this information is strictly
prohibited.
> > >> If
> > >>>>> you
> > >>>>>>> have received this communication in error, please contact
the
> > >> sender
> > >>>>> and
> > >>>>>>> delete the material from your computer.
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message