kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5061) client.id should be set for Connect producers/consumers
Date Sun, 24 Sep 2017 05:40:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178085#comment-16178085

ASF GitHub Bot commented on KAFKA-5061:

GitHub user Satyajitv opened a pull request:


    KAFKA-5061:clientid should be set for Connect Producers and Consumers

    Have added the new config in ConnectorConfig with priority "LOW", providing an option
to override client.id on a per-connector basis.
    And set client.id , using (worker group ID + task ID) by default, when client.id is not
provided in connector configuration, for distributed mode.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Satyajitv/kafka KAFKA-5061_clientid_should_be_set_for_Connect_Producers_and_Consumers

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3952
commit d9e200fed258fb2f6b9bcb0a2b2c876490a223cc
Author: satyajit <satyajit@telmate.com>
Date:   2017-09-24T02:54:01Z

    KAFKA-5061:clientid should be set for Connect Producers and Consumers


> client.id should be set for Connect producers/consumers
> -------------------------------------------------------
>                 Key: KAFKA-5061
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5061
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions:
>            Reporter: Ewen Cheslack-Postava
>              Labels: needs-kip, newbie++
> In order to properly monitor individual tasks using the producer and consumer metrics,
we need to have the framework disambiguate them. Currently when we create producers (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L362)
and create consumers (https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L371-L394)
the client ID is not being set. You can override it for the entire worker via worker-level
producer/consumer overrides, but you can't get per-task metrics.
> There are a couple of things we might want to consider doing here:
> 1. Provide default client IDs based on the worker group ID + task ID (providing uniqueness
for multiple connect clusters up to the scope of the Kafka cluster they are operating on).
This seems ideal since it's a good default; however it is a public-facing change and may need
a KIP. Normally I would be less worried about this, but some folks may be relying on picking
up metrics without this being set, in which case such a change would break their monitoring.
> 2. Allow overriding client.id on a per-connector basis. I'm not sure if this will really
be useful or not -- it lets you differentiate between metrics for different connectors' tasks,
but within a connector, all metrics would go to a single client.id. On the other hand, this
makes the tasks act as a single group from the perspective of broker handling of client IDs.

This message was sent by Atlassian JIRA

View raw message