flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Incompatible Apache Http lib in Flink kinesis connector
Date Tue, 20 Jun 2017 04:28:32 GMT
Hi,

We’ve seen this issue before [1]. The usual reason is that the httpcomponent dependencies
weren’t properly shaded in the flink-dist jar. Having them properly shaded should solve
the issue.

cc Bowen:
Are you building Flink yourself? If yes, what Maven version are you using? If you’re using
3.3.x+, after the first build under flink/, make sure to go to flink-dist/ and build a second
time for the dependencies to be properly shaded.
Alternatively, Maven 3.0.x+ is the recommended version, as 3.3.x has dependency shading issues.

If you’re not building Flink yourself, the cause could be that the Flink 1.3.0 flink-dist
jar wasn’t shaded properly, may need to double check.

Best,
Gordon

[1] https://issues.apache.org/jira/browse/FLINK-5013

On 20 June 2017 at 12:14:27 PM, Ted Yu (yuzhihong@gmail.com) wrote:

I logged FLINK-6951, referencing this thread.  

We can continue discussion there.  

Thanks  

On Mon, Jun 19, 2017 at 9:06 PM, Bowen Li <bowen.li@offerupnow.com> wrote:  

> Thanks, Ted! woo, this is unexpected. https://ci.apache.  
> org/projects/flink/flink-docs-release-1.3/setup/aws.html is out of date.  
>  
> I bet anyone using Kinesis with Flink will run into this issue. I can try  
> to build Flink myself and resolve this problem. But talking about a  
> feasible permanent solution for all flink-connector-kinesis users. Shall we  
> downgrade aws-java-sdk-kinesis version in flink-connector-kinesis, or shall  
> we upgrade httpcomponents version in Flink?  
>  
> Bowen  
>  
>  
> On Mon, Jun 19, 2017 at 7:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:  
>  
> > Here is the dependency in the flink-connector-kinesis module:  
> >  
> > [INFO] +- com.amazonaws:aws-java-sdk-kinesis:jar:1.10.71:compile  
> > [INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.71:compile  
> > [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile  
> > [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile  
> >  
> > Checking dependency tree of flink, the highest version is 4.2.x  
> >  
> > You can try building flink with dependency on 4.3.y of httpclient /  
> > httpcore  
> >  
> > FYI  
> >  
> >  
> >  
> > On Mon, Jun 19, 2017 at 4:52 PM, Bowen Li <bowen.li@offerupnow.com>  
> wrote:  
> >  
> > > Hi guys,  
> > > I'm trying to enable Flink's checkpoint on our Flink app. I got the  
> > > following Apache http jar compatibility error, and cannot figure out  
> how  
> > to  
> > > resolve it.  
> > >  
> > > Here's the stacktrace:  
> > >  
> > > ```  
> > >  
> > > 2017-06-19 15:07:39,828 INFO org.apache.flink.runtime.  
> taskmanager.Task  
> > > - Source: Custom Source -> (Timestamps/Watermarks,  
> Filter  
> > > -> Map, Filter -> Map, Filter -> Map) (1/1)  
> > > (37ab9429deda28e31fa0ed0ed1568654) switched from RUNNING to FAILED.  
> > > java.lang.NoSuchFieldError: INSTANCE  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.conn.  
> > > SdkConnectionKeepAliveStrategy.getKeepAliveDuration(  
> > > SdkConnectionKeepAliveStrategy.java:48)  
> > > at  
> > > org.apache.http.impl.client.DefaultRequestDirector.execute(  
> > > DefaultRequestDirector.java:535)  
> > > at  
> > > org.apache.http.impl.client.AbstractHttpClient.execute(  
> > > AbstractHttpClient.java:906)  
> > > at  
> > > org.apache.http.impl.client.AbstractHttpClient.execute(  
> > > AbstractHttpClient.java:805)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.  
> > > executeOneRequest(AmazonHttpClient.java:837)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.  
> > > executeHelper(AmazonHttpClient.java:607)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.  
> > > AmazonHttpClient.doExecute(AmazonHttpClient.java:376)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.  
> > > executeWithTimer(AmazonHttpClient.java:338)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.http.  
> > > AmazonHttpClient.execute(AmazonHttpClient.java:287)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.services.  
> > > kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1940)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.services.  
> > > kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1910)  
> > > at  
> > > org.apache.flink.kinesis.shaded.com.amazonaws.services.  
> > > kinesis.AmazonKinesisClient.describeStream(  
> AmazonKinesisClient.java:656)  
> > > at  
> > > org.apache.flink.streaming.connectors.kinesis.proxy.  
> > > KinesisProxy.describeStream(KinesisProxy.java:361)  
> > > at  
> > > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.  
> > > getShardsOfStream(KinesisProxy.java:323)  
> > > at  
> > > org.apache.flink.streaming.connectors.kinesis.proxy.  
> > > KinesisProxy.getShardList(KinesisProxy.java:231)  
> > > at  
> > > org.apache.flink.streaming.connectors.kinesis.internals.  
> > > KinesisDataFetcher.discoverNewShardsToSubscribe(  
> > > KinesisDataFetcher.java:430)  
> > > at  
> > > org.apache.flink.streaming.connectors.kinesis.  
> FlinkKinesisConsumer.run(  
> > > FlinkKinesisConsumer.java:202)  
> > > at  
> > > org.apache.flink.streaming.api.operators.StreamSource.  
> > > run(StreamSource.java:87)  
> > > at  
> > > org.apache.flink.streaming.api.operators.StreamSource.  
> > > run(StreamSource.java:55)  
> > > at  
> > > org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(  
> > > SourceStreamTask.java:95)  
> > > at  
> > > org.apache.flink.streaming.runtime.tasks.StreamTask.  
> > > invoke(StreamTask.java:262)  
> > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)  
> > > at java.lang.Thread.run(Thread.java:745)  
> > >  
> > > ```  
> > >  
> > > Here's my Flink environment setup:  
> > >  
> > > - I'm using flink-connector-kinesis_2.11-1.3.0.jar that I built  
> from  
> > > 1.3.0 source code.  
> > > - I followed  
> > > https://ci.apache.org/projects/flink/flink-docs-  
> > > release-1.3/setup/aws.html#flink-for-hadoop-27  
> > > and added all necessary dependency jars  
> > > - My application doesn't use Apache http/core.  
> > >  
> > >  
> > > Has anyone experienced the similar incompatibility issue?  
> > >  
> > > Thanks!  
> > > Bowen  
> > >  
> >  
>  

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message