Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Re: flink doesn't seem to serialize generated Avro pojo
From: Stefan Richter <s.richter@data-artisans.com>
In-Reply-To: <1498122582.900732.1017614808.781CE564@webmail.messagingengine.com>
Date: Thu, 22 Jun 2017 11:20:31 +0200
Cc: user@flink.apache.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <4F0F3964-D893-472E-8C8C-9BAAA227EB7A@data-artisans.com>
References: <1498122582.900732.1017614808.781CE564@webmail.messagingengine.com>
To: Bart van Deenen <bartvandeenen@fastmail.fm>
archived-at: Thu, 22 Jun 2017 09:20:46 -0000

Hi,

this =E2=80=9Eamazing error=E2=80=9C message typically means that the =
class of the object instance you created was loaded by a different =
classloader than the one that loaded the class in the code that tries to =
cast it. A class in Java is
fully identified by the canonical classname AND the classloader that =
loaded it. This makes it possible that two classes with the same name =
and bytecode are not instance of each other. Unfortunately, I have no =
concrete idea why that happens in your case,=20
but maybe this info helps to track down the problem.

Best,
Stefan

> Am 22.06.2017 um 11:09 schrieb Bart van Deenen =
<bartvandeenen@fastmail.fm>:
>=20
> Hi All
>=20
> I have a simple avro file=20
>=20
> {"namespace": "com.kpn.datalab.schemas.omnicrm.contact_history.v1",
> "type": "record",
> "name": "contactHistory",
> "fields": [
>     {"name": "events",     "type": {"type":"array",           =20
>     "items": "bytes"}},
>     {"name": "krn",      "type": "string"}
> ]
> }
>=20
> I generate a Java pojo (gist: https://goo.gl/FtM7T6) from this file =
via
>=20
> java -jar avro_tools.jar=20
> Version 1.7.7 of Apache Avro
>=20
> java -jar bin/avro_tools.jar compile schema <avrofile>
> generated/src/main/java
>=20
> This pojo file doesn't seem to want to be serialized by Flink, when I
> pack the compiled class file into a fat-jar job file.
>=20
> When I pass a certain byte array into the avro deserializer, this =
works
> fine in a regular Scala application, but when I do the same in a Flink
> job it bombs with a typecast exception.
>=20
> Scala code:
>=20
>      val payload: AnyRef =3D kafkaEvent.payload()   // this calls the
>      Avro deserializer: (gist: https://goo.gl/18UqJy)
>=20
>      println("canonical name: " + payload.getClass.getCanonicalName)
>      val chAvro =3D
>      try {
>        payload.asInstanceOf[contactHistory]
>      } catch {
>        case c:ClassCastException =3D>
>          println(c.getMessage)
>=20
> I get the following amazing error:
>=20
> "canonical name: " +
> com.kpn.datalab.schemas.omnicrm.contact_history.v1.contactHistory      =
 =20
> com.kpn.datalab.schemas.omnicrm.contact_history.v1.contactHistory =
cannot
> be cast to
> com.kpn.datalab.schemas.omnicrm.contact_history.v1.contactHistory
>=20
> I've verified the exact same code with the exact same byte-array input
> (checked by printing a hexdump from the Flink job), and the code =
itself
> is not the problem.
>=20
> If I put a jar containing the class file from the Pojo in the Flink =
lib
> directory, my Flink job works fine!
>=20
> Any ideas how I can put my Pojo in the fat jar, because I don't want =
to
> restart my Flink when I add new Avro schemas?
>=20
> Thanks
>=20
> Bart