apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandni Singh <chan...@datatorrent.com>
Subject Re: Stack overflow errors when launching job
Date Fri, 11 Mar 2016 23:46:43 GMT
Hey Ilya,

Can you please remove the duplicate output port from your implementation of
NewLineFileInputOperator as well.

Thanks,
Chandni

On Fri, Mar 11, 2016 at 3:42 PM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:

> Why do you want to have thread locality and partitioning together, you will
> lose parallel processing. What is the use case?
>
> Regards,
> Ashwin.
>
> On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ilya <
> Ilya.Ganelin@capitalone.com>
> wrote:
>
> > Now with files:
> > https://gist.github.com/ilganeli/7f770374113b40ffa18a
> >
> > From: "Ganelin, Ilya" <Ilya.Ganelin@capitalone.com<mailto:
> > Ilya.Ganelin@capitalone.com>>
> > Reply-To: "dev@apex.incubator.apache.org<mailto:
> > dev@apex.incubator.apache.org>" <dev@apex.incubator.apache.org<mailto:
> > dev@apex.incubator.apache.org>>
> > Date: Friday, March 11, 2016 at 3:02 PM
> > To: "dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org
> >"
> > <dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org>>
> > Subject: Stack overflow errors when launching job
> >
> > Hi guys – I’m running into a very frustrating issue where certain DAG
> > configurations cause the following error log (attached). When this
> happens,
> > my application even fails to launch. This does not seem to be a YARN
> issue
> > since this occurs even with a relatively small number of
> partitions/memory.
> >
> > I’ve attached the input and output operators in question.
> > I can get this to occur predictable by
> >
> >   1.  Increasing the partition count on my input operator (reads from
> > HDFS) - values above 20 cause this error
> >   2.  Increase the partition count on my output operator (writes to HDFS)
> > - values above 20 cause this error
> >   3.  Set stream locality from the default to either thread local, node
> > local, or container_local on the output operator
> >
> > This behavior is very frustrating as it’s preventing me from partitioning
> > my HDFS I/O appropriately, thus allowing me to scale to higher
> throughputs.
> >
> > Do you have any thoughts on what’s going wrong? I would love your
> feedback.
> >
> >
> > ________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> > ________________________________________________________
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message