apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Default App Master Container Memory Size
Date Mon, 14 Dec 2015 21:41:01 GMT
Hi Sandeep,

I have seen the issue even with small apps like Twitter Demo. It happens
less frequently in very small apps but the issue is still there. It happens
frequently in medium size apps (10 containers). Could we increase the
MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater
stability with the defaults settings without allocating too much additional
memory?

Thanks,
Tim

On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <sandeep@datatorrent.com>
wrote:

> With large number of containers, the problem becomes severe and I too have
> faced it.
>
> Not sure if default should be increased as that should work in most of the
> cases and even when new people are trying out Apex for the first time.
> They may not have enough memory.
>
> Regards
> Sandeep
> On 13 Dec 2015 11:30, "Ganelin, Ilya" <Ilya.Ganelin@capitalone.com> wrote:
>
> > Tim - great point, we reduce the memory consumption for operators and do
> > actually increase it for the app master, especially when we have more
> > operators. I misread your original email.
> >
> >
> >
> > Thank you,
> > Ilya Ganelin
> >
> >
> >
> > -----Original Message-----
> > From: Timothy Farkas [tim@datatorrent.com<mailto:tim@datatorrent.com>]
> > Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Default App Master Container Memory Size
> >
> >
> > Hi Ilya,
> >
> > Dou you reduce the memory to 256 mb for operators only or for the App
> > Master as well?
> >
> > Thanks,
> > Tim
> >
> > On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya <
> > Ilya.Ganelin@capitalone.com>
> > wrote:
> >
> > > Tim - in our applications we've actually tried to go with smaller
> > > containers, as small as 256 MB, simply because many operators don't
> > demand
> > > any memory and this allows us to run more partitions.
> > >
> > > We haven't noticed performance degradation.
> > >
> > >
> > >
> > > Thank you,
> > > Ilya Ganelin
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Timothy Farkas [tim@datatorrent.com<mailto:tim@datatorrent.com>]
> > > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time
> > > To: dev@apex.incubator.apache.org
> > > Subject: Default App Master Container Memory Size
> > >
> > >
> > > Hi All,
> > >
> > > The default value for MASTER_MEMORY_MB is 1024. I've observed in
> multiple
> > > demo apps and in the production apps of some of our users that 1024 MB
> is
> > > not enough memory, and the app master dies. This was observed on small
> > > applications with 5 - 20 containers. I think the default app master
> > memory
> > > size should be 2048.
> > >
> > > Any feedback?
> > >
> > > Thanks,
> > > Tim
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message