kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3854) Subsequent regex subscription calls fail
Date Mon, 04 Jul 2016 22:45:11 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ewen Cheslack-Postava updated KAFKA-3854:
    Fix Version/s:

> Subsequent regex subscription calls fail
> ----------------------------------------
>                 Key: KAFKA-3854
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3854
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: Vahid Hashemian
>            Assignee: Vahid Hashemian
>             Fix For:,
> There are a couple of issues with regex subscription in the new consumer:
> h6. Subsequent Subscriptions Fail
> When consecutive calls are made to new consumer's [regex subscription|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L850],
like below: 
> {code}
> consumer.subscribe(Pattern.compile("..."), listener);
> consumer.poll(0);
> consumer.subscribe(Pattern.compile("f.."), listener);
> consumer.poll(0);
> {code}
> the second call fails with the following error:
> {code}
> Exception in thread "main" java.lang.IllegalStateException: Subscription to topics, partitions
and pattern are mutually exclusive
> 	at org.apache.kafka.clients.consumer.internals.SubscriptionState.subscribe(SubscriptionState.java:175)
> 	at org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(KafkaConsumer.java:854)
> 	...
> {code}
> h6. Subsequent Subscriptions Fail to Trigger a Subscription Change and Rebalance
> Even after the code is tweaked to get around the above issue, only the first call to
regex subscription triggers an actual subscription and the subscription's rebalance listener.
The reason is the regex {{subscription}} / {{poll}} does not directly call {{changeSubscription(...)}}
in which {{needsPartitionAssignment}} is set and causes a rebalance. This method is called
only during the first regex {{subscription}} / {{poll}} when coordinator is unknown and [a
{{client.poll}} call|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractCoordinator.java#L179]
is made which eventually leads to [a {{changeSubscription}} call|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L161].
The subsequent call does not reach this point because the coordinator is already known.
> It seems due to the same reason, if a consumer is subscribed to a pattern, and later
on a new topic is created that matches that pattern, the consumer does not become subscribed

This message was sent by Atlassian JIRA

View raw message