Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.incubator.apache.org
Date: Wed, 23 Nov 2016 10:08:58 +0000 (UTC)
From: =?utf-8?Q?Jean-Baptiste_Onofr=C3=A9_=28JIRA=29?= <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.12990801.1468941789000.348851.1479895738639@Atlassian.JIRA>
In-Reply-To: <JIRA.12990801.1468941789000@Atlassian.JIRA>
References: <JIRA.12990801.1468941789000@Atlassian.JIRA> <JIRA.12990801.1468941789465@arcas>
Subject: [jira] [Commented] (BEAM-470) Spark Runner does not send the job
 execution information into the Spark History Server
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 23 Nov 2016 10:10:20 -0000


    [ https://issues.apache.org/jira/browse/BEAM-470?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D156896=
26#comment-15689626 ]=20

Jean-Baptiste Onofr=C3=A9 commented on BEAM-470:
-------------------------------------------

Thanks for the update, it's what I thought.

> Spark Runner does not send the job execution information into the Spark H=
istory Server
> -------------------------------------------------------------------------=
-------------
>
>                 Key: BEAM-470
>                 URL: https://issues.apache.org/jira/browse/BEAM-470
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>    Affects Versions: 0.2.0-incubating
>            Reporter: Isma=C3=ABl Mej=C3=ADa
>            Priority: Minor
>             Fix For: Not applicable
>
>
> If you run a Beam pipeline using the spark runner from spark (via spark-s=
ubmit), the execution is registered in the spark-history-server if it is ac=
tive and configured.
> if you do this directly from a main method with --runner=3DSparkRunner (t=
he beam way) the Beam runner does not report the execution to the history s=
erver, it seems the issue is the runner does not take into account an exist=
ing spark configuration file SPARK_HOME/conf/spark-defaults.conf (or there =
is not a way to tell the runner to take such conf into account).


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)