flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomasz Dobrzycki <dobrzycki.tom...@gmail.com>
Subject Re: Kinesis connector - Jackson issue
Date Thu, 28 Sep 2017 08:41:18 GMT
Hi guys,

I was able to solve the issue. I deleted all my Flink distributions
and followed these steps:

1) Clone Flink source (because I'm building Flink with Kinesis connector)
2) Checkout to release-1.3.1 (that's the version of Flink on EMR)
3) mvn clean install -Pinclude-kinesis -DskipTests (using Maven 3.2.5
for that so no need to do it in flink-dist again)
4) go to application folder
5) mvn clean package
- 'flink run' with file input works fine
- IntelliJ with file input works fine
- !! Caused by: java.lang.IllegalStateException: Socket not created by
this factory when running IntelliJ or 'flink run' with Kinesis input

6) follow the changes in
https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc
(need to apply them to pom.xml file of
flink-connectors/flink-connector-kinesis) !! important note !! follow
all of the including deletion of aws shading
- now the code should run via IntelliJ and 'flink run'

The trick was to apply all the changes from that PR.
Thanks for all the help and suggestions.
Oh, and Bowen Li - you're the champ mate, thanks ;)

Cheers,
Tomasz

On 26 September 2017 at 17:05, Tomasz Dobrzycki
<dobrzycki.tomasz@gmail.com> wrote:
> Yes I am using quickstart template. I have removed the exclusions for
> jackson: core, databind and annotations.
>
> On 26 September 2017 at 16:36, Tzu-Li (Gordon) Tai <tzulitai@apache.org> wrote:
>> Ah, I see.
>>
>> Are you using the Flink quickstart template to build your application?
>> I think exclusion is defined in the pom.xml of that archetype.
>>
>> Just above the exclusion I do see this message:
>> “WARNING: You have to remove these excludes if your code relies on other
>> version of these dependencies."
>>
>>
>> On 26 September 2017 at 5:27:47 PM, Tomasz Dobrzycki
>> (dobrzycki.tomasz@gmail.com) wrote:
>>
>> Hi Gordon,
>>
>> Thanks for your answer.
>> - I've built it with Maven 3.2.5
>> - I am using Jackson in my application (version 2.7.4)
>>
>> Something that I have noticed when building Kinesis connector is that
>> it excludes jackson:
>> [INFO] Excluding
>> com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.7.3
>> from the shaded jar.
>> even though I can't find any mention of that in it's pom.xml.
>>
>> Cheers,
>> Tomasz
>>
>> On 26 September 2017 at 15:43, Tzu-Li (Gordon) Tai <tzulitai@apache.org>
>> wrote:
>>> Hi Tomasz,
>>>
>>> Yes, dependency clashes may surface when executing actual job runs on
>>> clusters.
>>>
>>> A few things to probably check first:
>>> - Have you built Flink or the Kinesis connector with Maven version 3.3 or
>>> above? If yes, try using a lower version, as 3.3+ results in some shading
>>> issues when used to build Flink.
>>> - I’m not sure if the Kinesis client has a Jackson dependency, but you
>>> could
>>> also try checking if your application pulls in a conflicting Jackson
>>> version
>>> (with Flink, which uses 2.7.4) via some other dependency.
>>>
>>> Cheers,
>>> Gordon
>>>
>>>
>>> On 26 September 2017 at 4:28:27 PM, Tomasz Dobrzycki
>>> (dobrzycki.tomasz@gmail.com) wrote:
>>>
>>> Hi guys,
>>>
>>> I'm working with Kinesis connector and currently trying to solve a
>>> bizarre issue.
>>> I had problems with Kinesis and httpcomponents which I was able to
>>> solve using steps shown in:
>>>
>>> https://github.com/apache/flink/pull/4150/commits/9b539470ac308d7af9df9a70792aa1fa8c6995fc
>>>
>>> That did the trick and I am able to run my code successfully via
>>> IntelliJ. I am connecting to Kinesis stream hosted on AWS and reading
>>> messages just fine.
>>>
>>> Unfortunately that is not true for running Flink via command line
>>> scripts. I get this error when running start-local.sh into flink run
>>> ... :
>>> Caused by: java.lang.ClassNotFoundException:
>>> com.fasterxml.jackson.dataformat.cbor.CBORFactory
>>>
>>> I have built my Kinesis connector and installed it via mvn install. Am
>>> I missing some steps? I'm assuming that my code is fine given that I'm
>>> able to run it through IntelliJ.
>>>
>>> Anyone faced this problem or maybe some solution comes to your mind?
>>>
>>> Cheers
>>> Tomasz

Mime
View raw message