kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vahid Hashemian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3465) kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
Date Tue, 20 Jun 2017 21:43:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056517#comment-16056517

Vahid Hashemian commented on KAFKA-3465:

[~shanling2004] Sorry for the very very late response. I'm sure by now you've already figured
it out. As you guessed, {{ConsumerOffsetChecker}} does not support consumer groups based on
the new consumer. The consumer group tool ({{bin/kafka-consumer-groups.sh}}) should be used

> kafka.tools.ConsumerOffsetChecker won't align with kafka New Consumer mode
> --------------------------------------------------------------------------
>                 Key: KAFKA-3465
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3465
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions:
>            Reporter: BrianLing
> 1. When we enable mirrorMake to migrate Kafka event from one to other with "new.consumer"
> java -Xmx2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/kafka/kafka-app-logs -Dlog4j.configuration=file:/kafka/kafka_2.10-
-cp :/kafka/kafka_2.10-* -Dkafka.logs.filename=lvs-slca-mm.log kafka.tools.MirrorMaker
lvs-slca-mm.log --consumer.config ../config/consumer.properties --new.consumer --num.streams
4 --producer.config ../config/producer-slca.properties --whitelist risk.*
> 2. When we use ConsumerOffzsetChecker tool, notice the lag won't changed and the owner
is none.
> bin/kafka-run-class.sh  kafka.tools.ConsumerOffsetChecker --broker-info --group lvs.slca.mirrormaker
--zookeeper lvsdmetlvm01.lvs.paypal.com:2181 --topic XXXX
> Group           Topic                          Pid Offset          logSize         Lag
> lvs.slca.mirrormaker XXXX          0   418578332       418678347       100015       
> lvs.slca.mirrormaker XXXX         1   418598026       418698338       100312        
> [Root Cause]
> I think it's due to 0.9.0 new feature to switch zookeeper dependency to kafka internal
to store offset & consumer owner information. 
> 	Does it mean we can not use the below command to check new consumer’s lag since current
lag formula: lag= logSize – offset https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L80
> 	https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala#L174-L182
=> offSet Fetch from zookeeper instead of from Kafka

This message was sent by Atlassian JIRA

View raw message