flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hai Zhou UTC+8 (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-7964) Add Apache Kafka 1.0.0 connector
Date Thu, 02 Nov 2017 12:42:00 GMT
Hai Zhou UTC+8 created FLINK-7964:

             Summary: Add Apache Kafka 1.0.0 connector
                 Key: FLINK-7964
                 URL: https://issues.apache.org/jira/browse/FLINK-7964
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector
    Affects Versions: 1.4.0
            Reporter: Hai Zhou UTC+8
            Assignee: Hai Zhou UTC+8
            Priority: Major
             Fix For: 1.5.0

Kafka 1.0.0 is no mere bump of the version number. The Apache Kafka Project Management Committee
has packed a number of valuable enhancements into the release. Here is a summary of a few
of them:

* Since its introduction in version 0.10, the Streams API has become hugely popular among
Kafka users, including the likes of Pinterest, Rabobank, Zalando, and The New York Times.
In 1.0, the the API continues to evolve at a healthy pace. To begin with, the builder API
has been improved (KIP-120). A new API has been added to expose the state of active tasks
at runtime (KIP-130). The new cogroup API makes it much easier to deal with partitioned aggregates
with fewer StateStores and fewer moving parts in your code (KIP-150). Debuggability gets easier
with enhancements to the print() and writeAsText() methods (KIP-160). And if that’s not
enough, check out KIP-138 and KIP-161 too. For more on streams, check out the Apache Kafka
Streams documentation, including some helpful new tutorial videos.
* Operating Kafka at scale requires that the system remain observable, and to make that easier,
we’ve made a number of improvements to metrics. These are too many to summarize without
becoming tedious, but Connect metrics have been significantly improved (KIP-196), a litany
of new health check metrics are now exposed (KIP-188), and we now have a global topic and
partition count (KIP-168). Check out KIP-164 and KIP-187 for even more.
* We now support Java 9, leading, among other things, to significantly faster TLS and CRC32C
implementations. Over-the-wire encryption will be faster now, which will keep Kafka fast and
compute costs low when encryption is enabled.
* In keeping with the security theme, KIP-152 cleans up the error handling on Simple Authentication
Security Layer (SASL) authentication attempts. Previously, some authentication error conditions
were indistinguishable from broker failures and were not logged in a clear way. This is cleaner
* Kafka can now tolerate disk failures better. Historically, JBOD storage configurations have
not been recommended, but the architecture has nevertheless been tempting: after all, why
not rely on Kafka’s own replication mechanism to protect against storage failure rather
than using RAID? With KIP-112, Kafka now handles disk failure more gracefully. A single disk
failure in a JBOD broker will not bring the entire broker down; rather, the broker will continue
serving any log files that remain on functioning disks.
* Since release 0.11.0, the idempotent producer (which is the producer used in the presence
of a transaction, which of course is the producer we use for exactly-once processing) required
max.in.flight.requests.per.connection to be equal to one. As anyone who has written or tested
a wire protocol can attest, this put an upper bound on throughput. Thanks to KAFKA-5949, this
can now be as large as five, relaxing the throughput constraint quite a bit.

This message was sent by Atlassian JIRA

View raw message