beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juan Carlos Garcia <jcgarc...@gmail.com>
Subject Re: Beam SparkRunner and Spark KryoSerializer problem
Date Mon, 30 Jul 2018 13:15:10 GMT
Hi Jean,

Thanks for taking a look.


On Mon, Jul 30, 2018 at 2:49 PM, Jean-Baptiste Onofré <jb@nanthrax.net>
wrote:

> Hi Juan,
>
> it seems that has been introduce by the metrics layer in the core runner
> API.
>
> Let me check.
>
> Regards
> JB
>
> On 30/07/2018 14:47, Juan Carlos Garcia wrote:
> > Bump!
> >
> > Does any of the core-dev roam around here?
> >
> > Can someone provide a feedback about BEAM-4597
> > <https://issues.apache.org/jira/browse/BEAM-4597>
> >
> > Thanks and regards,
> >
> > On Thu, Jul 19, 2018 at 3:41 PM, Juan Carlos Garcia <jcgarciam@gmail.com
> > <mailto:jcgarciam@gmail.com>> wrote:
> >
> >     Folks,
> >
> >     Its someone using the SparkRunner out there with the Spark
> >     KryoSerializer ?
> >
> >     We are being force to use the not so efficient 'JavaSerializer' with
> >     Spark because we face the following exception:
> >
> >     <exception>
> >     Exception in thread "main" java.lang.RuntimeException:
> >     org.apache.spark.SparkException: Job aborted due to stage failure:
> >     Exception while getting task result:
> >     com.esotericsoftware.kryo.KryoException: Unable to find class:
> >     org.apache.beam.runners.core.metrics.MetricsContainerImpl$$
> Lambda$31/1875283985
> >     Serialization trace:
> >     factory (org.apache.beam.runners.core.metrics.MetricsMap)
> >     counters (org.apache.beam.runners.core.metrics.MetricsContainerImpl)
> >     metricsContainers
> >     (org.apache.beam.runners.core.metrics.MetricsContainerStepMap)
> >     metricsContainers
> >     (org.apache.beam.runners.spark.io.SparkUnboundedSource$Metadata)
> >     at
> >     org.apache.beam.runners.spark.SparkPipelineResult.
> runtimeExceptionFrom(SparkPipelineResult.java:55)
> >     at
> >     org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(
> SparkPipelineResult.java:72)
> >     at
> >     org.apache.beam.runners.spark.SparkPipelineResult.access$
> 000(SparkPipelineResult.java:41)
> >     at
> >     org.apache.beam.runners.spark.SparkPipelineResult$
> StreamingMode.stop(SparkPipelineResult.java:163)
> >     at
> >     org.apache.beam.runners.spark.SparkPipelineResult.offerNewState(
> SparkPipelineResult.java:198)
> >     at
> >     org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(
> SparkPipelineResult.java:101)
> >     at
> >     org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(
> SparkPipelineResult.java:87)
> >     at
> >     org.apache.beam.examples.BugWithKryoOnSpark.main(
> BugWithKryoOnSpark.java:75)
> >     </exception>
> >
> >     I created a jira ticket and attached a project example on it,
> >     https://issues.apache.org/jira/browse/BEAM-4597
> >     <https://issues.apache.org/jira/browse/BEAM-4597>
> >
> >     Any feedback is appreciated.
> >
> >     --
> >
> >     JC
> >
> >
> >
> >
> > --
> >
> > JC
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>



-- 

JC

Mime
View raw message