apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chandni Singh <chan...@datatorrent.com>
Subject Re: Stack overflow errors when launching job
Date Fri, 11 Mar 2016 23:18:59 GMT
Hey Ilya,

Can you please assign more memory to the App Master and check?

Chandni

On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com>
wrote:

> Now with files:
> https://gist.github.com/ilganeli/7f770374113b40ffa18a
>
> From: "Ganelin, Ilya" <Ilya.Ganelin@capitalone.com<mailto:
> Ilya.Ganelin@capitalone.com>>
> Reply-To: "dev@apex.incubator.apache.org<mailto:
> dev@apex.incubator.apache.org>" <dev@apex.incubator.apache.org<mailto:
> dev@apex.incubator.apache.org>>
> Date: Friday, March 11, 2016 at 3:02 PM
> To: "dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org>"
> <dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org>>
> Subject: Stack overflow errors when launching job
>
> Hi guys – I’m running into a very frustrating issue where certain DAG
> configurations cause the following error log (attached). When this happens,
> my application even fails to launch. This does not seem to be a YARN issue
> since this occurs even with a relatively small number of partitions/memory.
>
> I’ve attached the input and output operators in question.
> I can get this to occur predictable by
>
>   1.  Increasing the partition count on my input operator (reads from
> HDFS) - values above 20 cause this error
>   2.  Increase the partition count on my output operator (writes to HDFS)
> - values above 20 cause this error
>   3.  Set stream locality from the default to either thread local, node
> local, or container_local on the output operator
>
> This behavior is very frustrating as it’s preventing me from partitioning
> my HDFS I/O appropriately, thus allowing me to scale to higher throughputs.
>
> Do you have any thoughts on what’s going wrong? I would love your feedback.
>
>
> ________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message