Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
Date: Fri, 18 Mar 2016 00:13:33 +0000 (UTC)
From: "Ilya Ganelin (JIRA)" <jira@apache.org>
To: dev@apex.incubator.apache.org
Message-ID: <JIRA.12950993.1458161633000.54716.1458260013563@Atlassian.JIRA>
In-Reply-To: <JIRA.12950993.1458161633000@Atlassian.JIRA>
References: <JIRA.12950993.1458161633000@Atlassian.JIRA>
 <JIRA.12950993.1458161633087@arcas>
Subject: [jira] [Comment Edited] (APEXCORE-392) Stack Overflow when
 launching jobs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/APEXCORE-392?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15=
200660#comment-15200660 ]=20

Ilya Ganelin edited comment on APEXCORE-392 at 3/18/16 12:13 AM:
-----------------------------------------------------------------

I've updated the gist with a more complete example that reliably causes thi=
s failure. The code is not a complete project because other components of t=
he project include code that cannot be shared.=20

An example that causes this to break:
{{code}}
// Add our operators to the DAG
NewlineFileInputOperator fileInputOperator =3D
dag.addOperator("NewLines", NewlineFileInputOperator.class);
fileInputOperator.setCoreSite(conf.get("coreSite"));
fileInputOperator.setHdfsSite(conf.get("hdfsSite"));
fileInputOperator.setDirectory(conf.get("citadel.data.rawAuths"));

setPartitioner(dag, fileInputOperator, conf.getInt("citadel.numHdfsPartitio=
ns", 16));

String label =3D "HDHT";
final HdfsFileOutputOperator latenciesOutput =3D
addHdfsOutputOp(dag, "LatencyOut" + label, "latencies", conf);
setPartitioner(dag, latenciesOutput, conf.getInt("citadel.numHdfsPartitions=
", 16));

dag.addStream("latencies2File" + label, fileInputOperator.output, latencies=
Output.input);
{{code}}
    <property>
        <name>citadel.numPartitions</name>
        <value>20</value>
    </property>

    <property>
        <name>citadel.numHdfsPartitions</name>
        <value>20</value>
    </property>


was (Author: ilganeli):
I've updated the gist with a more complete example that reliably causes thi=
s failure. The code is not a complete project because other components of t=
he project include code that cannot be shared.=20

An example that causes this to break:
{ code }
// Add our operators to the DAG
NewlineFileInputOperator fileInputOperator =3D
dag.addOperator("NewLines", NewlineFileInputOperator.class);
fileInputOperator.setCoreSite(conf.get("coreSite"));
fileInputOperator.setHdfsSite(conf.get("hdfsSite"));
fileInputOperator.setDirectory(conf.get("citadel.data.rawAuths"));

setPartitioner(dag, fileInputOperator, conf.getInt("citadel.numHdfsPartitio=
ns", 16));

String label =3D "HDHT";
final HdfsFileOutputOperator latenciesOutput =3D
addHdfsOutputOp(dag, "LatencyOut" + label, "latencies", conf);
setPartitioner(dag, latenciesOutput, conf.getInt("citadel.numHdfsPartitions=
", 16));

dag.addStream("latencies2File" + label, fileInputOperator.output, latencies=
Output.input);
{ code }
    <property>
        <name>citadel.numPartitions</name>
        <value>20</value>
    </property>

    <property>
        <name>citadel.numHdfsPartitions</name>
        <value>20</value>
    </property>

> Stack Overflow when launching jobs
> ----------------------------------
>
>                 Key: APEXCORE-392
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-392
>             Project: Apache Apex Core
>          Issue Type: Bug
>    Affects Versions: 3.2.0, 3.3.0
>            Reporter: Ilya Ganelin
>            Priority: Blocker
>
> I=E2=80=99m running into a very frustrating issue where certain DAG confi=
gurations cause the following error log (attached). When this happens, my a=
pplication even fails to launch. This does not seem to be a YARN issue sinc=
e this occurs even with a relatively small number of partitions/memory.=20
> This issue DOES appear to be related to HDFS input/output operations sinc=
e the specific parameter that appears to affect things is the number of phy=
sical partitions for the HDFS input/output operators.
> I=E2=80=99ve also attached the input and output operators in question:
> https://gist.github.com/ilganeli/7f770374113b40ffa18a
> I can get this to occur predictable by
>   1.  Increasing the partition count on my input operator (reads from HDF=
S) - values above 20 cause this error
>   2.  Increase the partition count on my output operator (writes to HDFS)=
 - values above 20 cause this error
>   3.  Set stream locality from the default to either thread local, node l=
ocal, or container_local on the output operator
> This behavior is very frustrating as it=E2=80=99s preventing me from part=
itioning my HDFS I/O appropriately, thus allowing me to scale to higher thr=
oughputs.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)