flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shannon Carey <sca...@expedia.com>
Subject Re: AWS exception serialization problem
Date Sat, 11 Mar 2017 01:53:26 GMT
Here ya go (see attached).

From: Robert Metzger <rmetzger@apache.org<mailto:rmetzger@apache.org>>
Date: Friday, March 10, 2017 at 1:18 PM
To: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>>
Subject: Re: AWS exception serialization problem

Can one of you guys provide us with a minimal example to reproduce the issue? (Ideally locally,
not using EMR?)
I think once we can reproduce the issue its easy to fix.

On Thu, Mar 9, 2017 at 1:24 AM, Bruno Aranda <brunoaranda@gmail.com<mailto:brunoaranda@gmail.com>>

Hi Stephan, we are running Flink 1.2.0 on Yarn (AWS EMR cluster)

On Wed, 8 Mar 2017, 21:41 Stephan Ewen, <sewen@apache.org<mailto:sewen@apache.org>>
@Bruno: How are you running Flink? On yarn, standalone, mesos, docker?

On Wed, Mar 8, 2017 at 2:13 PM, Bruno Aranda <brunoaranda@gmail.com<mailto:brunoaranda@gmail.com>>

We have seen something similar in Flink 1.2. We have an operation that parses some JSON, and
when it fails to parse it, we can see the ClassNotFoundException for the relevant exception
(in our case JsResultException from the play-json library). The library is indeed in the shaded
JAR, otherwise we would not be able to parse the JSON.



On Wed, 8 Mar 2017 at 12:57 Tzu-Li (Gordon) Tai <tzulitai@apache.org<mailto:tzulitai@apache.org>>
Hi Shannon,

Just to clarify:

From the error trace, it seems like that the messages fetched from Kafka are serialized `AmazonS3Exception`s,
and you’re emitting a stream of `AmazonS3Exception` as records from FlinkKafkaConsumer?
Is this correct? If so, I think we should just make sure that the `com.amazonaws.services.s3.model.AmazonS3Exception`
class exists in the user fat jar.

Also, what is the Flink version you are using?


View raw message