flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adebski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7133) Fix Elasticsearch version interference
Date Fri, 07 Jul 2017 18:15:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16078480#comment-16078480

Adebski commented on FLINK-7133:

I'm the person that described the details of the issue on the mailing list.

[~tzulitai] The situation is exactly as you've described: the version of ES configured in
elasticsearch-base module is 1.7.1, maven shade plugin detects that this module has ASM as
dependency and the shades it under org.apache.flink.shaded package. I've only tested it in
local environment because I'm evaluating Flink so it is easier for me to execute my programs
directly from IDEA instead of deploying to local cluster. It may work on clustered environment
because (correct me if I am wrong) when job is executed in Flink it has a separate classloader
in runtime so shaded ASM in version 5.0.4 is available to Flink internal classes and does
not clash with 4.X version from ES connector.

I could work on this issue but we would have to decide the proper course of action. Either
remove base module and duplicate the classes or somehow configure build process to not shade
ASM in elasticsearch-base module (I've tried to do that but I'm not that proficient in Maven).

> Fix Elasticsearch version interference
> --------------------------------------
>                 Key: FLINK-7133
>                 URL: https://issues.apache.org/jira/browse/FLINK-7133
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 1.3.0, 1.3.1
>            Reporter: Aljoscha Krettek
>            Priority: Blocker
>             Fix For: 1.3.2
> At least two users have encountered problems with shading in the Elasticsearch connector:
>  - https://lists.apache.org/thread.html/b5bc1f690dc894ea9a8b69e82c89eb89ba6dfc2fec2588d2ccacee2c@%3Cuser.flink.apache.org%3E
>  - https://lists.apache.org/thread.html/2356670d168f61c20e34611e3c4aeb9c9b3f959f23a9833f631da1ba@%3Cuser.flink.apache.org%3E
> The problem seems to be (quote from the second mail):
> {code}
> I've found out the source of the problem when I build flink locally.
> elastic-search base depends on (by default) ES version 1.7.1 that depends on
> asm 4.1 and that version is shaded to elasticsearch-base-jar. I tried to set
> elasticsearch.version property in Maven to 5.1.2 (the same as elasticsearch5
> connector) but then elasticsearch-base does not compile:
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project flink-connector-elasticsearch-base_2.11:
> Compilation failure
> /home/adebski/Downloads/flink-release-1.3.1/flink-connectors/flink-connector-elasticsearch-base/src/test/java/org/apache/flink/streaming/connectors/elasticsearch/ElasticsearchSinkBaseTest.java:[491,92]
> no suitable constructor found for
> BulkItemResponse(int,java.lang.String,org.elasticsearch.action.ActionResponse)
> [ERROR] constructor
> org.elasticsearch.action.bulk.BulkItemResponse.BulkItemResponse(int,java.lang.String,org.elasticsearch.action.DocWriteResponse)
> is not applicable
> [ERROR] (argument mismatch; org.elasticsearch.action.ActionResponse cannot
> be converted to org.elasticsearch.action.DocWriteResponse)
> [ERROR] constructor
> org.elasticsearch.action.bulk.BulkItemResponse.BulkItemResponse(int,java.lang.String,org.elasticsearch.action.bulk.BulkItemResponse.Failure)
> is not applicable
> [ERROR] (argument mismatch; org.elasticsearch.action.ActionResponse cannot
> be converted to org.elasticsearch.action.bulk.BulkItemResponse.Failure)
> {code}
> To me, it seems like we have to get rid of the "base" package and have two completely
separate packages.

This message was sent by Atlassian JIRA

View raw message