kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rhauch <...@git.apache.org>
Subject [GitHub] kafka pull request #2984: KAFKA-4667 Connect uses AdminClient to create inte...
Date Fri, 05 May 2017 17:58:16 GMT
GitHub user rhauch opened a pull request:

    https://github.com/apache/kafka/pull/2984

    KAFKA-4667 Connect uses AdminClient to create internal topics when needed

    The backing store for offsets, status, and configs now attempts to use the new AdminClient
to look up the internal topics and create them if they don’t yet exist. If the necessary
APIs are not available in the connected broker, the stores fall back to the old behavior of
relying upon auto-created topics. Kafka Connect requires a minimum of Apache Kafka 0.10.0.1-cp1,
and the AdminClient can work with all versions since 0.10.0.0.
    
    All three of Connect’s internal topics are created as compacted topics, and new distributed
worker configuration properties control the replication factor for all three topics and the
number of partitions for the offsets and status topics; the config topic requires a single
partition and does not allow it to be set via configuration. All of these new configuration
properties have sensible defaults, meaning users can upgrade without having to change any
of the existing configurations. In most situations, existing Connect deployments will have
already created the storage topics prior to upgrading.
    
    The replication factor defaults to 3, so anyone running Kafka clusters with fewer nodes
than 3 will receive an error unless they explicitly set the replication factor for the three
internal topics. This is actually desired behavior, since it signals the users that they should
be aware they are not using sufficient replication for production use.
    
    The integration tests use a cluster with a single broker, so they were changed to explicitly
specify a replication factor of 1 and a single partition.
    
    The `KafkaAdminClientTest` was refactored to extract a utility for setting up a `KafkaAdminClient`
with a `MockClient` for unit tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rhauch/kafka kafka-4667

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/2984.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2984
    
----
commit ea181ba29a678e47a5ff88ca0ad4c995aa4e017b
Author: Randall Hauch <rhauch@gmail.com>
Date:   2017-05-05T17:14:07Z

    KAFKA-4667 Connect can create internal topics using AdminClient
    
    The backing store for offsets, status, and configs now attempts to use the new AdminClient
to look up the
    internal topics and create them if they don’t yet exist. If the necessary APIs are not
available in the connected
    broker, the stores fall back to the old behavior of relying upon auto-created topics.
Kafka Connect requires
    a minimum of Apache Kafka 0.10.0.1-cp1, and the AdminClient can work with all versions
since 0.10.0.0.
    
    All three of Connect’s internal topics are created as compacted topics, and new distributed
worker configuration
    properties control the replication factor for all three topics and the number of partitions
for the offsets
    and status topics; the config topic requires a single partition and does not allow it
to be set via configuration.
    All of these new configuration properties have sensible defaults, meaning users can upgrade
without having to
    change any of the existing configurations. In most situations, existing Connect deployments
will have already
    created the storage topics prior to upgrading.
    
    The replication factor defaults to 3, so anyone running Kafka clusters with fewer nodes
than 3 will receive an
    error unless they explicitly set the replication factor for the three internal topics.
This is actually desired behavior,
    since it signals the users that they should be aware they are not using sufficient replication
for production use.
    
    The integration tests use a cluster with a single broker, so they were changed to explictly
specify a replication factor of 1
    and a single partition.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message