apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chinmay Kolhatkar <chin...@datatorrent.com>
Subject Re: Default App Master Container Memory Size
Date Thu, 24 Dec 2015 05:53:23 GMT
I agree with Gaurav.
We should try to identify what is causing current memory limit not to work.

If there is a need for a stop gap solution, we could temporarily make it
higher. But I believe this is not a solution unless we answer the "why"
question.


~ Chinmay.

On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

> Tim,
>
> This setting used to work fine earlier. What did we change/add in App
> master that is causing this behavior. We should know the reason before
> bumping the memory. May be there is some other issue.
>
> Thanks
> - Gaurav
>
> > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <sandeep@datatorrent.com>
> wrote:
> >
> > I am ok with 1280MB.
> > On 24 Dec 2015 06:34, "Timothy Farkas" <tim@datatorrent.com> wrote:
> >
> >> If there are no objections I will open a pull request shortly to bump up
> >> the default App Master Memory to 1280 MB. I recently got burned by this
> >> default AGAIN when the App Master randomly died in the middle of demo.
> The
> >> demo application was small and only had 6 physical operators.
> >>
> >> Thanks,
> >> Tim
> >>
> >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <ram@datatorrent.com
> >
> >> wrote:
> >>
> >>> Sorry, meant to say 4-6GB, not MB.
> >>>
> >>> Ram
> >>>
> >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <
> ram@datatorrent.com>
> >>> wrote:
> >>>
> >>>> I'm OK with increasing it to 1280MB.
> >>>>
> >>>> I agree with Sandeep about new people trying out Apex,
> >>>> often on limited environments like VMs limited to 4-6MB. With a
> default
> >>> of
> >>>> 0.5GB per Buffer Server
> >>>> and 1GB per operator, things start to fail when the user has more than
> >> a
> >>>> handful of operators.
> >>>>
> >>>> I'm guessing it's not just the number of the operators that puts
> memory
> >>>> pressure on master memory
> >>>> but the size of the jars that get pulled in by all the operator
> classes
> >>>> and their dependencies. If this is
> >>>> true, one could cause failure even with a single, sufficiently complex
> >>>> operator.
> >>>>
> >>>> Ram
> >>>>
> >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <tim@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Sandeep,
> >>>>>
> >>>>> I have seen the issue even with small apps like Twitter Demo. It
> >> happens
> >>>>> less frequently in very small apps but the issue is still there.
It
> >>>>> happens
> >>>>> frequently in medium size apps (10 containers). Could we increase
the
> >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
> >>>>> stability with the defaults settings without allocating too much
> >>>>> additional
> >>>>> memory?
> >>>>>
> >>>>> Thanks,
> >>>>> Tim
> >>>>>
> >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <
> >>>>> sandeep@datatorrent.com>
> >>>>> wrote:
> >>>>>
> >>>>>> With large number of containers, the problem becomes severe
and I
> >> too
> >>>>> have
> >>>>>> faced it.
> >>>>>>
> >>>>>> Not sure if default should be increased as that should work
in most
> >> of
> >>>>> the
> >>>>>> cases and even when new people are trying out Apex for the first
> >> time.
> >>>>>> They may not have enough memory.
> >>>>>>
> >>>>>> Regards
> >>>>>> Sandeep
> >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <Ilya.Ganelin@capitalone.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Tim - great point, we reduce the memory consumption for
operators
> >>> and
> >>>>> do
> >>>>>>> actually increase it for the app master, especially when
we have
> >>> more
> >>>>>>> operators. I misread your original email.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thank you,
> >>>>>>> Ilya Ganelin
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> >>> tim@datatorrent.com
> >>>>>> ]
> >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard
Time
> >>>>>>> To: dev@apex.incubator.apache.org
> >>>>>>> Subject: Re: Default App Master Container Memory Size
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi Ilya,
> >>>>>>>
> >>>>>>> Dou you reduce the memory to 256 mb for operators only or
for the
> >>> App
> >>>>>>> Master as well?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Tim
> >>>>>>>
> >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> >>>>>>> Ilya.Ganelin@capitalone.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Tim - in our applications we've actually tried to go
with
> >> smaller
> >>>>>>>> containers, as small as 256 MB, simply because many
operators
> >>> don't
> >>>>>>> demand
> >>>>>>>> any memory and this allows us to run more partitions.
> >>>>>>>>
> >>>>>>>> We haven't noticed performance degradation.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Thank you,
> >>>>>>>> Ilya Ganelin
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Timothy Farkas [tim@datatorrent.com<mailto:
> >>>>> tim@datatorrent.com>]
> >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard
Time
> >>>>>>>> To: dev@apex.incubator.apache.org
> >>>>>>>> Subject: Default App Master Container Memory Size
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi All,
> >>>>>>>>
> >>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've
observed in
> >>>>>> multiple
> >>>>>>>> demo apps and in the production apps of some of our
users that
> >>> 1024
> >>>>> MB
> >>>>>> is
> >>>>>>>> not enough memory, and the app master dies. This was
observed on
> >>>>> small
> >>>>>>>> applications with 5 - 20 containers. I think the default
app
> >>> master
> >>>>>>> memory
> >>>>>>>> size should be 2048.
> >>>>>>>>
> >>>>>>>> Any feedback?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Tim
> >>>>>>>> ________________________________________________________
> >>>>>>>>
> >>>>>>>> The information contained in this e-mail is confidential
and/or
> >>>>>>>> proprietary to Capital One and/or its affiliates and
may only be
> >>>>> used
> >>>>>>>> solely in performance of work or services for Capital
One. The
> >>>>>>> information
> >>>>>>>> transmitted herewith is intended only for use by the
individual
> >> or
> >>>>>> entity
> >>>>>>>> to which it is addressed. If the reader of this message
is not
> >> the
> >>>>>>> intended
> >>>>>>>> recipient, you are hereby notified that any review,
> >>> retransmission,
> >>>>>>>> dissemination, distribution, copying or other use of,
or taking
> >> of
> >>>>> any
> >>>>>>>> action in reliance upon this information is strictly
prohibited.
> >>> If
> >>>>> you
> >>>>>>>> have received this communication in error, please contact
the
> >>> sender
> >>>>>> and
> >>>>>>>> delete the material from your computer.
> >>>>>>>>
> >>>>>>> ________________________________________________________
> >>>>>>>
> >>>>>>> The information contained in this e-mail is confidential
and/or
> >>>>>>> proprietary to Capital One and/or its affiliates and may
only be
> >>> used
> >>>>>>> solely in performance of work or services for Capital One.
The
> >>>>>> information
> >>>>>>> transmitted herewith is intended only for use by the individual
or
> >>>>> entity
> >>>>>>> to which it is addressed. If the reader of this message
is not the
> >>>>>> intended
> >>>>>>> recipient, you are hereby notified that any review,
> >> retransmission,
> >>>>>>> dissemination, distribution, copying or other use of, or
taking of
> >>> any
> >>>>>>> action in reliance upon this information is strictly prohibited.
> >> If
> >>>>> you
> >>>>>>> have received this communication in error, please contact
the
> >> sender
> >>>>> and
> >>>>>>> delete the material from your computer.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message