flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bowen Li <bowen...@offerupnow.com>
Subject Re: Incompatible Apache Http lib in Flink kinesis connector
Date Tue, 20 Jun 2017 04:06:56 GMT
Thanks, Ted! woo, this is unexpected. https://ci.apache.
org/projects/flink/flink-docs-release-1.3/setup/aws.html is out of date.

I bet anyone using Kinesis with Flink will run into this issue. I can try
to build Flink myself and resolve this problem. But talking about a
feasible permanent solution for all flink-connector-kinesis users. Shall we
downgrade aws-java-sdk-kinesis version in flink-connector-kinesis, or shall
we upgrade httpcomponents version in Flink?

Bowen


On Mon, Jun 19, 2017 at 7:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Here is the dependency in the flink-connector-kinesis module:
>
> [INFO] +- com.amazonaws:aws-java-sdk-kinesis:jar:1.10.71:compile
> [INFO] |  \- com.amazonaws:aws-java-sdk-core:jar:1.10.71:compile
> [INFO] |     +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile
> [INFO] |     +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile
>
> Checking dependency tree of flink, the highest version is 4.2.x
>
> You can try building flink with dependency on 4.3.y of httpclient /
> httpcore
>
> FYI
>
>
>
> On Mon, Jun 19, 2017 at 4:52 PM, Bowen Li <bowen.li@offerupnow.com> wrote:
>
> > Hi guys,
> >     I'm trying to enable Flink's checkpoint on our Flink app. I got the
> > following Apache http jar compatibility error, and cannot figure out how
> to
> > resolve it.
> >
> >     Here's the stacktrace:
> >
> >         ```
> >
> > 2017-06-19 15:07:39,828 INFO  org.apache.flink.runtime.taskmanager.Task
> >                 - Source: Custom Source -> (Timestamps/Watermarks, Filter
> > -> Map, Filter -> Map, Filter -> Map) (1/1)
> > (37ab9429deda28e31fa0ed0ed1568654) switched from RUNNING to FAILED.
> > java.lang.NoSuchFieldError: INSTANCE
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.http.conn.
> > SdkConnectionKeepAliveStrategy.getKeepAliveDuration(
> > SdkConnectionKeepAliveStrategy.java:48)
> > at
> > org.apache.http.impl.client.DefaultRequestDirector.execute(
> > DefaultRequestDirector.java:535)
> > at
> > org.apache.http.impl.client.AbstractHttpClient.execute(
> > AbstractHttpClient.java:906)
> > at
> > org.apache.http.impl.client.AbstractHttpClient.execute(
> > AbstractHttpClient.java:805)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.
> > executeOneRequest(AmazonHttpClient.java:837)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.
> > executeHelper(AmazonHttpClient.java:607)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.http.
> > AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.
> > executeWithTimer(AmazonHttpClient.java:338)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.http.
> > AmazonHttpClient.execute(AmazonHttpClient.java:287)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.services.
> > kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1940)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.services.
> > kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1910)
> > at
> > org.apache.flink.kinesis.shaded.com.amazonaws.services.
> > kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:656)
> > at
> > org.apache.flink.streaming.connectors.kinesis.proxy.
> > KinesisProxy.describeStream(KinesisProxy.java:361)
> > at
> > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.
> > getShardsOfStream(KinesisProxy.java:323)
> > at
> > org.apache.flink.streaming.connectors.kinesis.proxy.
> > KinesisProxy.getShardList(KinesisProxy.java:231)
> > at
> > org.apache.flink.streaming.connectors.kinesis.internals.
> > KinesisDataFetcher.discoverNewShardsToSubscribe(
> > KinesisDataFetcher.java:430)
> > at
> > org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.run(
> > FlinkKinesisConsumer.java:202)
> > at
> > org.apache.flink.streaming.api.operators.StreamSource.
> > run(StreamSource.java:87)
> > at
> > org.apache.flink.streaming.api.operators.StreamSource.
> > run(StreamSource.java:55)
> > at
> > org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(
> > SourceStreamTask.java:95)
> > at
> > org.apache.flink.streaming.runtime.tasks.StreamTask.
> > invoke(StreamTask.java:262)
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > ```
> >
> >       Here's my Flink environment setup:
> >
> >    - I'm using flink-connector-kinesis_2.11-1.3.0.jar that I built from
> >    1.3.0 source code.
> >    - I followed
> >    https://ci.apache.org/projects/flink/flink-docs-
> > release-1.3/setup/aws.html#flink-for-hadoop-27
> >    and added all necessary dependency jars
> >    - My application doesn't use Apache http/core.
> >
> >
> >      Has anyone experienced the similar incompatibility issue?
> >
> > Thanks!
> > Bowen
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message