apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashwin Chandra Putta <ashwinchand...@gmail.com>
Subject Re: Reg container getting killed without throwing exceptions
Date Wed, 20 Jan 2016 23:56:57 GMT
Venkatesh,

If you do not see the window id moving forward, it usually means that the
business logic is blocking the operator. Please check if window id is
moving forward.

Regards,
Ashwin.

On Wed, Jan 20, 2016 at 3:37 PM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

> Venkatesh,
>
> I think actually issue is that operator is getting blocked as you
> mentioned that operator is taking too long to process and it is not showing
> any processed and emitted tuples. AM is not getting any heart beat from
> operator so it kills it.
>
> Thanks
> - Gaurav
>
> > On Jan 20, 2016, at 3:17 PM, Kottapalli, Venkatesh
> <VKottapalli@DIRECTV.com> wrote:
> >
> > Thanks for your inputs Gaurav and Tim.
> >
> > When it is OOM, I see it in the container logs but it in this case I
> don’t find any.
> >
> > I see the processing part in the operator running and printing logs
> without any issues end to end but not reaching the end window. It might be
> because of the grouping logic that we have added  in the end window that is
> causing OOM but the container logs doesn’t show it.
> >
> > The operator is taking long to process.  Total processed and emitted by
> that operator is always 0.
> >
> > I shall try to increase memory on the Application master and the
> container as well and see if it works else I will try on a smaller load and
> see if it is a scaling issue because of OOM.
> >
> > Right now, I don’t have access to the AM logs.
> >
> >
> > Regards,
> > Venkatesh.
> >
> > -----Original Message-----
> > From: Timothy Farkas [mailto:tim@datatorrent.com]
> > Sent: Wednesday, January 20, 2016 3:11 PM
> > To: dev@apex.incubator.apache.org
> > Subject: Re: Reg container getting killed without throwing exceptions
> >
> > Hey Venkatesh,
> >
> > How much memory is allocated to the App Master? You should allocate
> atleast 2GB to app master with this property.
> >
> >
> >  <property>
> >    <name>dt.attr.MASTER_MEMORY_MB</name>
> >    <value>2048</value>
> >  </property>
> >
> > Otherwise the App Master may die suddenly without printing anything to
> logs.
> >
> > Thanks,
> > Tim
> >
> > On Wed, Jan 20, 2016 at 2:47 PM, Gaurav Gupta <gaurav@datatorrent.com>
> > wrote:
> >
> >> Venkatesh,
> >>
> >> Did you see any OOM exception? It would be good to see the AM logs and
> >> container logs to find out more.
> >>
> >> Thanks
> >> - Gaurav
> >>
> >>> On Jan 20, 2016, at 2:42 PM, Kottapalli, Venkatesh <
> >> VKottapalli@directv.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>>               I get the following message when the container is
> >>> getting
> >> killed. I don't find logs for any exceptions being thrown. How do we
> >> identify the root cause for this issue?
> >>> Sorry for being very abstract.
> >>>
> >>> Container killed by the ApplicationMaster.
> >>> Container killed on request. Exit code is 143 Container exited with
> >>> a non-zero exit code 143
> >>>
> >>> Regards,
> >>> Venkatesh.
> >>
> >>
>
>


-- 

Regards,
Ashwin.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message