flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vidhu5269 <...@git.apache.org>
Subject [GitHub] flink issue #4015: [FLINK-6301] [flink-connector-kafka-0.10] Upgrading Kafka...
Date Mon, 03 Jul 2017 19:36:28 GMT
Github user vidhu5269 commented on the issue:

    Hi @tzulitai 
    Apologies for such a long delay. It took me quite a while to come back to this.
    I ran the updated connector on the cluster and didn't see any dependency conflicts. The
job using it was reading from a gzipped avro topic and producing into two different topics:
a text and an avro topic. Both the consumption and production worked as expected.
    It was done on a standalone cluster with 2 workers and 1 master with 8 slots/worker. The
job was using 10 slots and was running on the both the workers. Each worker had 8 vCPUs and
8 GB of RAM. 
    With this job, I also verified that there wasn't any memory leak with the `kafka-clients`
version change.
    Following is the diff between the master and version change commit from `mvn dependency:tree`:
    < [INFO] +- org.apache.kafka:kafka-clients:jar:
    > [INFO] +- org.apache.kafka:kafka-clients:jar:
    < [INFO] +- org.apache.kafka:kafka_2.11:jar:
    < [INFO] |  +- com.101tec:zkclient:jar:0.8:test
    > [INFO] +- org.apache.kafka:kafka_2.11:jar:
    > [INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.3:test
    < [INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:test
    < [INFO] |  +- net.sf.jopt-simple:jopt-simple:jar:4.9:test
    < [INFO] |  \- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
    < [INFO] |     +- jline:jline:jar:0.9.94:provided
    < [INFO] |     \- io.netty:netty:jar:3.7.0.Final:provided
    > [INFO] |  +- com.101tec:zkclient:jar:0.10:test
    > [INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:provided
    > [INFO] |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:test
    > [INFO] |  |  +- io.netty:netty:jar:3.8.0.Final:provided
    From what we can see here, apart from the new kafka-clients library, there are a few changes
coming from its tests-jar as well.
    1. The tests-jar has new version of `net.sf.jopt-simple:jopt-simple` and `com.101tec:zkclient:jar:0.10`
in `test` scope. 
    2. `jline:jline` is not there in the new version.
    3. `3.8.0-Final` of `io.netty:netty` is being pulled in (from `flakka-remote`) instead
of `3.7.0-Final`. 
    Since the job and the unit tests worked as expected, I am assuming that these dependency
changes are not breaking anything. Do tell me if I am missing something here.
    Although, these are not new dependencies but I still verified that `zkclient` and `netty`
both have ASL 2.0 whereas, jopt-simple has `MIT` which is also compatible with ASL 2.0. So,
we should be good as far as licensing goes.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message