Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Thu, 3 Aug 2017 09:47:00 +0000 (UTC)
From: =?utf-8?Q?Jean-Baptiste_Onofr=C3=A9_=28JIRA=29?= <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13092147.1501753394000.83044.1501753620306@Atlassian.JIRA>
In-Reply-To: <JIRA.13092147.1501753394000@Atlassian.JIRA>
References: <JIRA.13092147.1501753394000@Atlassian.JIRA> <JIRA.13092147.1501753394706@jira-lw-us.apache.org>
Subject: [jira] [Assigned] (BEAM-2719) Beam job hangs at Evaluating
 ParMultiDo when submitted via spark-runner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Thu, 03 Aug 2017 09:47:06 -0000


     [ https://issues.apache.org/jira/browse/BEAM-2719?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jean-Baptiste Onofr=C3=A9 reassigned BEAM-2719:
------------------------------------------

    Assignee: Jean-Baptiste Onofr=C3=A9  (was: Amit Sela)

> Beam job hangs at Evaluating ParMultiDo when submitted via spark-runner=
=20
> ------------------------------------------------------------------------
>
>                 Key: BEAM-2719
>                 URL: https://issues.apache.org/jira/browse/BEAM-2719
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>    Affects Versions: 2.0.0
>         Environment: OSX / i5 / 10GB
>            Reporter: Sathish Jayaraman
>            Assignee: Jean-Baptiste Onofr=C3=A9
>
> Hi,
> The Beam job submitted for execution via spark-submit does not get past t=
he Evaluating ParMultiDo step. The compile execution runs fine when given -=
-runner=3DSparkRunner as parameter. But if I bundle the jar & submit it usi=
ng spark-submit, there were no executors getting assigned. I tried to submi=
t with both master spark-url & YARN but no luck in getting it executed past=
 that step. Below is the command I used to submit & job log from YARN.=20
> I tried executing in both local single node cluster & in Azure HDInsight =
cluster, the result is the same. So I guess there is nothing wrong in the S=
park configuration & could be a bug.=20
> {code}
> $ ~/spark/bin/spark-submit --class org.apache.beam.examples.WordCount --m=
aster yarn --executor-memory 2G --num-executors 2 target/word-count-beam-0.=
1-shaded.jar --runner=3DSparkRunner --inputFile=3Dpom.xml --output=3Dcounts
> {code}
> {code}
> 17/08/03 13:00:33 INFO client.RMProxy: Connecting to ResourceManager at /=
0.0.0.0:8030
> 17/08/03 13:00:33 INFO yarn.YarnRMClient: Registering the ApplicationMast=
er
> 17/08/03 13:00:34 INFO yarn.YarnAllocator: Will request 2 executor contai=
ner(s), each with 1 core(s) and 2432 MB memory (including 384 MB of overhea=
d)
> 17/08/03 13:00:34 INFO yarn.YarnAllocator: Submitted 2 unlocalized contai=
ner requests.
> 17/08/03 13:00:34 INFO yarn.ApplicationMaster: Started progress reporter =
thread with (heartbeat : 3000, initial allocation : 200) intervals
> 17/08/03 13:00:35 INFO impl.AMRMClientImpl: Received new token for : 192.=
168.0.7:50173
> 17/08/03 13:00:35 INFO yarn.YarnAllocator: Launching container container_=
1501744514957_0003_01_000002 on host 192.168.0.7
> 17/08/03 13:00:35 INFO yarn.YarnAllocator: Received 1 containers from YAR=
N, launching executors on 1 of them.
> 17/08/03 13:00:35 INFO impl.ContainerManagementProtocolProxy: yarn.client=
.max-cached-nodemanagers-proxies : 0
> 17/08/03 13:00:35 INFO impl.ContainerManagementProtocolProxy: Opening pro=
xy : 192.168.0.7:50173
> 17/08/03 13:00:37 INFO yarn.YarnAllocator: Launching container container_=
1501744514957_0003_01_000003 on host 192.168.0.7
> 17/08/03 13:00:37 INFO yarn.YarnAllocator: Received 1 containers from YAR=
N, launching executors on 1 of them.
> 17/08/03 13:00:37 INFO impl.ContainerManagementProtocolProxy: yarn.client=
.max-cached-nodemanagers-proxies : 0
> 17/08/03 13:00:37 INFO impl.ContainerManagementProtocolProxy: Opening pro=
xy : 192.168.0.7:50173
> {code}


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)