Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Tue, 4 Apr 2017 09:01:41 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13056830.1489704654000.198862.1491296501600@Atlassian.JIRA>
In-Reply-To: <JIRA.13056830.1489704654000@Atlassian.JIRA>
References: <JIRA.13056830.1489704654000@Atlassian.JIRA> <JIRA.13056830.1489704654854@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-1737) Interpreting a Single-output ParDo
 as a Multi-output ParDo with a single output causes serialization failures
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 04 Apr 2017 09:01:47 -0000


    [ https://issues.apache.org/jira/browse/BEAM-1737?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15954=
857#comment-15954857 ]=20

ASF GitHub Bot commented on BEAM-1737:
--------------------------------------

GitHub user amitsela opened a pull request:

    https://github.com/apache/beam/pull/2420

    [BEAM-1737] Interpreting a Single-output ParDo as a Multi-output ParDo =
with a single output causes serialization failuresremove use of EvaluationC=
ontext in DStream lambda, it is not serializ=E2=80=A6

    =E2=80=A6able and also redundant in this
   =20
    case.
   =20
    Be sure to do all of the following to help us incorporate your contribu=
tion
    quickly and easily:
   =20
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enabl=
e
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.or=
g/licenses/icla.pdf).
   =20
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/amitsela/beam BEAM-1737

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/2420.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2420
   =20
----
commit c4f12af13889112cff28f21ba5173d6514c07cf0
Author: Amit Sela <amitsela33@gmail.com>
Date:   2017-04-04T08:46:38Z

    remove use of EvaluationContext in DStream lambda, it is not serializab=
le and also redundant in this
    case.

----


> Interpreting a Single-output ParDo as a Multi-output ParDo with a single =
output causes serialization failures
> -------------------------------------------------------------------------=
------------------------------------
>
>                 Key: BEAM-1737
>                 URL: https://issues.apache.org/jira/browse/BEAM-1737
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Thomas Groh
>            Priority: Minor
>
> This is the cause of having a separate path and implementation for single=
-output ParDos, even though both go through the same translator.
> Partial stacktrace:
> Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 5.946 sec=
 <<< FAILURE! - in org.apache.beam.runners.spark.translation.streaming.Crea=
teStreamTest                                         [8233/41535]
> testLateDataAccumulating(org.apache.beam.runners.spark.translation.stream=
ing.CreateStreamTest)  Time elapsed: 3.593 sec  <<< ERROR!
> java.lang.RuntimeException:=20
> java.io.NotSerializableException: DStream checkpointing has been enabled =
but the DStreams with their functions are not serializable
> org.apache.beam.runners.spark.translation.EvaluationContext
> Serialization stack:
>         - object not serializable (class: org.apache.beam.runners.spark.t=
ranslation.EvaluationContext, value: org.apache.beam.runners.spark.translat=
ion.EvaluationContext@a8c55d7)
>         - field (class: org.apache.beam.runners.spark.translation.streami=
ng.StreamingTransformTranslator$8$1, name: val$context, type: class org.apa=
che.beam.runners.spark.translation.EvaluationContext)
>         - object (class org.apache.beam.runners.spark.translation.streami=
ng.StreamingTransformTranslator$8$1, org.apache.beam.runners.spark.translat=
ion.streaming.StreamingTransformTranslator$8$1@44f50940)
>         - field (class: org.apache.spark.streaming.api.java.JavaDStreamLi=
ke$$anonfun$transformToPair$1, name: transformFunc$3, type: interface org.a=
pache.spark.api.java.function.Function)
>         - object (class org.apache.spark.streaming.api.java.JavaDStreamLi=
ke$$anonfun$transformToPair$1, <function1>)
>         - field (class: org.apache.spark.streaming.dstream.DStream$$anonf=
un$transform$1$$anonfun$apply$21, name: cleanedF$2, type: interface scala.F=
unction1)
>         - object (class org.apache.spark.streaming.dstream.DStream$$anonf=
un$transform$1$$anonfun$apply$21, <function2>)
>         - field (class: org.apache.spark.streaming.dstream.DStream$$anonf=
un$transform$2$$anonfun$5, name: cleanedF$3, type: interface scala.Function=
2)
>         - object (class org.apache.spark.streaming.dstream.DStream$$anonf=
un$transform$2$$anonfun$5, <function2>)
>         - writeObject data (class: org.apache.spark.streaming.dstream.DSt=
reamCheckpointData)
>         - object (class org.apache.spark.streaming.dstream.DStreamCheckpo=
intData, [
> 0 checkpoint files=20
> ])
> ....
>         at org.apache.beam.runners.spark.SparkPipelineResult.runtimeExcep=
tionFrom(SparkPipelineResult.java:60)
>         at org.apache.beam.runners.spark.SparkPipelineResult.beamExceptio=
nFrom(SparkPipelineResult.java:77)
>         at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFin=
ish(SparkPipelineResult.java:113)
>         at org.apache.beam.runners.spark.TestSparkRunner.awaitWatermarksO=
rTimeout(TestSparkRunner.java:195)
>         at org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRun=
ner.java:127)
>         at org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRun=
ner.java:82)
>         at org.apache.beam.sdk.Pipeline.run(Pipeline.java:210)
> ...


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)