flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Incompatible Apache Http lib in Flink kinesis connector
Date Tue, 20 Jun 2017 04:14:16 GMT
I logged FLINK-6951, referencing this thread.

We can continue discussion there.

Thanks

On Mon, Jun 19, 2017 at 9:06 PM, Bowen Li <bowen.li@offerupnow.com> wrote:

> Thanks, Ted! woo, this is unexpected. https://ci.apache.
> org/projects/flink/flink-docs-release-1.3/setup/aws.html is out of date.
>
> I bet anyone using Kinesis with Flink will run into this issue. I can try
> to build Flink myself and resolve this problem. But talking about a
> feasible permanent solution for all flink-connector-kinesis users. Shall we
> downgrade aws-java-sdk-kinesis version in flink-connector-kinesis, or shall
> we upgrade httpcomponents version in Flink?
>
> Bowen
>
>
> On Mon, Jun 19, 2017 at 7:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Here is the dependency in the flink-connector-kinesis module:
> >
> > [INFO] +- com.amazonaws:aws-java-sdk-kinesis:jar:1.10.71:compile
> > [INFO] |  \- com.amazonaws:aws-java-sdk-core:jar:1.10.71:compile
> > [INFO] |     +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile
> > [INFO] |     +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile
> >
> > Checking dependency tree of flink, the highest version is 4.2.x
> >
> > You can try building flink with dependency on 4.3.y of httpclient /
> > httpcore
> >
> > FYI
> >
> >
> >
> > On Mon, Jun 19, 2017 at 4:52 PM, Bowen Li <bowen.li@offerupnow.com>
> wrote:
> >
> > > Hi guys,
> > >     I'm trying to enable Flink's checkpoint on our Flink app. I got the
> > > following Apache http jar compatibility error, and cannot figure out
> how
> > to
> > > resolve it.
> > >
> > >     Here's the stacktrace:
> > >
> > >         ```
> > >
> > > 2017-06-19 15:07:39,828 INFO  org.apache.flink.runtime.
> taskmanager.Task
> > >                 - Source: Custom Source -> (Timestamps/Watermarks,
> Filter
> > > -> Map, Filter -> Map, Filter -> Map) (1/1)
> > > (37ab9429deda28e31fa0ed0ed1568654) switched from RUNNING to FAILED.
> > > java.lang.NoSuchFieldError: INSTANCE
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.conn.
> > > SdkConnectionKeepAliveStrategy.getKeepAliveDuration(
> > > SdkConnectionKeepAliveStrategy.java:48)
> > > at
> > > org.apache.http.impl.client.DefaultRequestDirector.execute(
> > > DefaultRequestDirector.java:535)
> > > at
> > > org.apache.http.impl.client.AbstractHttpClient.execute(
> > > AbstractHttpClient.java:906)
> > > at
> > > org.apache.http.impl.client.AbstractHttpClient.execute(
> > > AbstractHttpClient.java:805)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.
> > > executeOneRequest(AmazonHttpClient.java:837)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.
> > > executeHelper(AmazonHttpClient.java:607)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.
> > > AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.
> > > executeWithTimer(AmazonHttpClient.java:338)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.
> > > AmazonHttpClient.execute(AmazonHttpClient.java:287)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.services.
> > > kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1940)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.services.
> > > kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1910)
> > > at
> > > org.apache.flink.kinesis.shaded.com.amazonaws.services.
> > > kinesis.AmazonKinesisClient.describeStream(
> AmazonKinesisClient.java:656)
> > > at
> > > org.apache.flink.streaming.connectors.kinesis.proxy.
> > > KinesisProxy.describeStream(KinesisProxy.java:361)
> > > at
> > > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.
> > > getShardsOfStream(KinesisProxy.java:323)
> > > at
> > > org.apache.flink.streaming.connectors.kinesis.proxy.
> > > KinesisProxy.getShardList(KinesisProxy.java:231)
> > > at
> > > org.apache.flink.streaming.connectors.kinesis.internals.
> > > KinesisDataFetcher.discoverNewShardsToSubscribe(
> > > KinesisDataFetcher.java:430)
> > > at
> > > org.apache.flink.streaming.connectors.kinesis.
> FlinkKinesisConsumer.run(
> > > FlinkKinesisConsumer.java:202)
> > > at
> > > org.apache.flink.streaming.api.operators.StreamSource.
> > > run(StreamSource.java:87)
> > > at
> > > org.apache.flink.streaming.api.operators.StreamSource.
> > > run(StreamSource.java:55)
> > > at
> > > org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(
> > > SourceStreamTask.java:95)
> > > at
> > > org.apache.flink.streaming.runtime.tasks.StreamTask.
> > > invoke(StreamTask.java:262)
> > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > ```
> > >
> > >       Here's my Flink environment setup:
> > >
> > >    - I'm using flink-connector-kinesis_2.11-1.3.0.jar that I built
> from
> > >    1.3.0 source code.
> > >    - I followed
> > >    https://ci.apache.org/projects/flink/flink-docs-
> > > release-1.3/setup/aws.html#flink-for-hadoop-27
> > >    and added all necessary dependency jars
> > >    - My application doesn't use Apache http/core.
> > >
> > >
> > >      Has anyone experienced the similar incompatibility issue?
> > >
> > > Thanks!
> > > Bowen
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message