flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai" <tzuli...@apache.org>
Subject Re: Incompatible Apache Http lib in Flink kinesis connector
Date Tue, 20 Jun 2017 05:26:47 GMT
Hi Bowen,

Thanks for the info. I checked the 1.3.0 release jars, and they do not have unshaded httpcomponent
dependencies, so that shouldn’t be the problem.

Looking back into the stack trace you posted, the conflict seems to be a different problem.
The conflict seems to be with clashes with the aws-java-sdk version, and not the httpcomponent
dependency.
The “INSTANCE” field actually does exist in the aws-java-sdk version that the Kinesis
connector is using.

Could it be that you have other conflicting aws-java-sdk versions in your jar?

Cheers,
Gordon

On 20 June 2017 at 12:55:17 PM, Bowen Li (bowen.li@offerupnow.com) wrote:

Hi Gordon,  
Here's what I use:  

- Flink: I didn't build Flink myself. I download  
http://apache.mirrors.lucidnetworks.net/flink/flink-1.3.0/flink-1.3.0-bin-hadoop27-scala_2.11.tgz
 
from https://flink.apache.org/downloads.html (Hadoop® 2.7, Scala 2.11)  
- flink-kinesis-connector: I  
build flink-connector-kinesis_2.11-1.3.0.jar myself, from source code  
downloaded at *#Source* section in  
https://flink.apache.org/downloads.html.  
- Mvn -v: Apache Maven 3.2.5  


In short, I didn't build Flink. Most likely that dependencies in  
either flink-dist or flink-kinesis-connector is not shaded properly?  

Thanks!  
Bowen  

On Mon, Jun 19, 2017 at 9:28 PM, Tzu-Li (Gordon) Tai <tzulitai@apache.org>  
wrote:  

> Hi,  
>  
> We’ve seen this issue before [1]. The usual reason is that the  
> httpcomponent dependencies weren’t properly shaded in the flink-dist jar.  
> Having them properly shaded should solve the issue.  
>  
> cc Bowen:  
> Are you building Flink yourself? If yes, what Maven version are you using?  
> If you’re using 3.3.x+, after the first build under flink/, make sure to go  
> to flink-dist/ and build a second time for the dependencies to be properly  
> shaded.  
> Alternatively, Maven 3.0.x+ is the recommended version, as 3.3.x has  
> dependency shading issues.  
>  
> If you’re not building Flink yourself, the cause could be that the Flink  
> 1.3.0 flink-dist jar wasn’t shaded properly, may need to double check.  
>  
> Best,  
> Gordon  
>  
> [1] https://issues.apache.org/jira/browse/FLINK-5013  
>  
> On 20 June 2017 at 12:14:27 PM, Ted Yu (yuzhihong@gmail.com) wrote:  
>  
> I logged FLINK-6951, referencing this thread.  
>  
> We can continue discussion there.  
>  
> Thanks  
>  
> On Mon, Jun 19, 2017 at 9:06 PM, Bowen Li <bowen.li@offerupnow.com> wrote:  
>  
> > Thanks, Ted! woo, this is unexpected. https://ci.apache.  
> > org/projects/flink/flink-docs-release-1.3/setup/aws.html is out of date.  
> >  
> > I bet anyone using Kinesis with Flink will run into this issue. I can try  
> > to build Flink myself and resolve this problem. But talking about a  
> > feasible permanent solution for all flink-connector-kinesis users. Shall  
> we  
> > downgrade aws-java-sdk-kinesis version in flink-connector-kinesis, or  
> shall  
> > we upgrade httpcomponents version in Flink?  
> >  
> > Bowen  
> >  
> >  
> > On Mon, Jun 19, 2017 at 7:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:  
> >  
> > > Here is the dependency in the flink-connector-kinesis module:  
> > >  
> > > [INFO] +- com.amazonaws:aws-java-sdk-kinesis:jar:1.10.71:compile  
> > > [INFO] | \- com.amazonaws:aws-java-sdk-core:jar:1.10.71:compile  
> > > [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile  
> > > [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile  
> > >  
> > > Checking dependency tree of flink, the highest version is 4.2.x  
> > >  
> > > You can try building flink with dependency on 4.3.y of httpclient /  
> > > httpcore  
> > >  
> > > FYI  
> > >  
> > >  
> > >  
> > > On Mon, Jun 19, 2017 at 4:52 PM, Bowen Li <bowen.li@offerupnow.com> 

> > wrote:  
> > >  
> > > > Hi guys,  
> > > > I'm trying to enable Flink's checkpoint on our Flink app. I got the  
> > > > following Apache http jar compatibility error, and cannot figure out 

> > how  
> > > to  
> > > > resolve it.  
> > > >  
> > > > Here's the stacktrace:  
> > > >  
> > > > ```  
> > > >  
> > > > 2017-06-19 15:07:39,828 INFO org.apache.flink.runtime.  
> > taskmanager.Task  
> > > > - Source: Custom Source -> (Timestamps/Watermarks,  
> > Filter  
> > > > -> Map, Filter -> Map, Filter -> Map) (1/1)  
> > > > (37ab9429deda28e31fa0ed0ed1568654) switched from RUNNING to FAILED.  
> > > > java.lang.NoSuchFieldError: INSTANCE  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.conn.  
> > > > SdkConnectionKeepAliveStrategy.getKeepAliveDuration(  
> > > > SdkConnectionKeepAliveStrategy.java:48)  
> > > > at  
> > > > org.apache.http.impl.client.DefaultRequestDirector.execute(  
> > > > DefaultRequestDirector.java:535)  
> > > > at  
> > > > org.apache.http.impl.client.AbstractHttpClient.execute(  
> > > > AbstractHttpClient.java:906)  
> > > > at  
> > > > org.apache.http.impl.client.AbstractHttpClient.execute(  
> > > > AbstractHttpClient.java:805)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient. 

> > > > executeOneRequest(AmazonHttpClient.java:837)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient. 

> > > > executeHelper(AmazonHttpClient.java:607)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.  
> > > > AmazonHttpClient.doExecute(AmazonHttpClient.java:376)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient. 

> > > > executeWithTimer(AmazonHttpClient.java:338)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.http.  
> > > > AmazonHttpClient.execute(AmazonHttpClient.java:287)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.services.  
> > > > kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1940)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.services.  
> > > > kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1910)  
> > > > at  
> > > > org.apache.flink.kinesis.shaded.com.amazonaws.services.  
> > > > kinesis.AmazonKinesisClient.describeStream(  
> > AmazonKinesisClient.java:656)  
> > > > at  
> > > > org.apache.flink.streaming.connectors.kinesis.proxy.  
> > > > KinesisProxy.describeStream(KinesisProxy.java:361)  
> > > > at  
> > > > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.  
> > > > getShardsOfStream(KinesisProxy.java:323)  
> > > > at  
> > > > org.apache.flink.streaming.connectors.kinesis.proxy.  
> > > > KinesisProxy.getShardList(KinesisProxy.java:231)  
> > > > at  
> > > > org.apache.flink.streaming.connectors.kinesis.internals.  
> > > > KinesisDataFetcher.discoverNewShardsToSubscribe(  
> > > > KinesisDataFetcher.java:430)  
> > > > at  
> > > > org.apache.flink.streaming.connectors.kinesis.  
> > FlinkKinesisConsumer.run(  
> > > > FlinkKinesisConsumer.java:202)  
> > > > at  
> > > > org.apache.flink.streaming.api.operators.StreamSource.  
> > > > run(StreamSource.java:87)  
> > > > at  
> > > > org.apache.flink.streaming.api.operators.StreamSource.  
> > > > run(StreamSource.java:55)  
> > > > at  
> > > > org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(  
> > > > SourceStreamTask.java:95)  
> > > > at  
> > > > org.apache.flink.streaming.runtime.tasks.StreamTask.  
> > > > invoke(StreamTask.java:262)  
> > > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)  
> > > > at java.lang.Thread.run(Thread.java:745)  
> > > >  
> > > > ```  
> > > >  
> > > > Here's my Flink environment setup:  
> > > >  
> > > > - I'm using flink-connector-kinesis_2.11-1.3.0.jar that I built  
> > from  
> > > > 1.3.0 source code.  
> > > > - I followed  
> > > > https://ci.apache.org/projects/flink/flink-docs-  
> > > > release-1.3/setup/aws.html#flink-for-hadoop-27  
> > > > and added all necessary dependency jars  
> > > > - My application doesn't use Apache http/core.  
> > > >  
> > > >  
> > > > Has anyone experienced the similar incompatibility issue?  
> > > >  
> > > > Thanks!  
> > > > Bowen  
> > > >  
> > >  
> >  
>  

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message