flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vinoyang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7964) Add Apache Kafka 1.0/1.1 connectors
Date Fri, 10 Aug 2018 08:47:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575955#comment-16575955

vinoyang commented on FLINK-7964:

[~twalthr] Agree Timo, I am processing Kafka 2.0 connector which tracked by issue FLINK-9697.

> Add Apache Kafka 1.0/1.1 connectors
> -----------------------------------
>                 Key: FLINK-7964
>                 URL: https://issues.apache.org/jira/browse/FLINK-7964
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>    Affects Versions: 1.4.0
>            Reporter: Hai Zhou
>            Assignee: Hai Zhou
>            Priority: Major
>             Fix For: 1.7.0
> Kafka 1.0.0 is no mere bump of the version number. The Apache Kafka Project Management
Committee has packed a number of valuable enhancements into the release. Here is a summary
of a few of them:
> * Since its introduction in version 0.10, the Streams API has become hugely popular among
Kafka users, including the likes of Pinterest, Rabobank, Zalando, and The New York Times.
In 1.0, the the API continues to evolve at a healthy pace. To begin with, the builder API
has been improved (KIP-120). A new API has been added to expose the state of active tasks
at runtime (KIP-130). The new cogroup API makes it much easier to deal with partitioned aggregates
with fewer StateStores and fewer moving parts in your code (KIP-150). Debuggability gets easier
with enhancements to the print() and writeAsText() methods (KIP-160). And if that’s not
enough, check out KIP-138 and KIP-161 too. For more on streams, check out the Apache Kafka
Streams documentation, including some helpful new tutorial videos.
> * Operating Kafka at scale requires that the system remain observable, and to make that
easier, we’ve made a number of improvements to metrics. These are too many to summarize
without becoming tedious, but Connect metrics have been significantly improved (KIP-196),
a litany of new health check metrics are now exposed (KIP-188), and we now have a global topic
and partition count (KIP-168). Check out KIP-164 and KIP-187 for even more.
> * We now support Java 9, leading, among other things, to significantly faster TLS and
CRC32C implementations. Over-the-wire encryption will be faster now, which will keep Kafka
fast and compute costs low when encryption is enabled.
> * In keeping with the security theme, KIP-152 cleans up the error handling on Simple
Authentication Security Layer (SASL) authentication attempts. Previously, some authentication
error conditions were indistinguishable from broker failures and were not logged in a clear
way. This is cleaner now.
> * Kafka can now tolerate disk failures better. Historically, JBOD storage configurations
have not been recommended, but the architecture has nevertheless been tempting: after all,
why not rely on Kafka’s own replication mechanism to protect against storage failure rather
than using RAID? With KIP-112, Kafka now handles disk failure more gracefully. A single disk
failure in a JBOD broker will not bring the entire broker down; rather, the broker will continue
serving any log files that remain on functioning disks.
> * Since release 0.11.0, the idempotent producer (which is the producer used in the presence
of a transaction, which of course is the producer we use for exactly-once processing) required
max.in.flight.requests.per.connection to be equal to one. As anyone who has written or tested
a wire protocol can attest, this put an upper bound on throughput. Thanks to KAFKA-5949, this
can now be as large as five, relaxing the throughput constraint quite a bit.

This message was sent by Atlassian JIRA

View raw message