apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chaitanya (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2518) Kafka input operator stops reading tuples when there is a UNKNOWN_MEMBER_ID error during committed offset processing
Date Mon, 10 Jul 2017 07:24:00 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079967#comment-16079967
] 

Chaitanya commented on APEXMALHAR-2518:
---------------------------------------

[~pramod.immaneni] Could you please share the complete stack trace ? 
Also, Please go through the below links, looks like a similar issue:
* https://issues.apache.org/jira/browse/KAFKA-2985
* https://stackoverflow.com/questions/38394662/error-unknown-member-id-occurred-while-committing-offsets-for-group-xxx

> Kafka input operator stops reading tuples when there is a UNKNOWN_MEMBER_ID error during
committed offset processing
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: APEXMALHAR-2518
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2518
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: Pramod Immaneni
>            Assignee: Pramod Immaneni
>
> Kafka 0.9 operator stores offsets that are completely processed and no longer needed
(committed offsets) back in kafka. It does so by making a kafka API call. If the response
from kafka server to this call comes back with an UNKNOWN_MEMBER_ID error, it results in the
kafka consumer state changing to needing partition re-assignment and no further messages are
returned by the consumer. There are a couple of other errors that result in the same state
including when rebalance is in progress.
> What exactly caused this error is not known but the following is the likely reason due
to the conditions surrounding the application. When the operator has temporarily stalled due
to back-pressure exerted by the slow downstream, it will eventually stall the operator kafka
consumer thread that is reading messages from kafka. This will result in the thread not making
any kafka consumer API calls and it will result in no heartbeats being sent to kafka server.
This can cause the server to evict the consumer after a timeout period. This could have been
the cause for the UNKNOWN_MEMBER_ID error.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message