apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2518) Kafka input operator stops reading tuples when there is a UNKNOWN_MEMBER_ID error during committed offset processing
Date Fri, 14 Jul 2017 18:03:00 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087701#comment-16087701

ASF GitHub Bot commented on APEXMALHAR-2518:

sandeshh commented on issue #644: APEXMALHAR-2518 Terminating operator when there is a server
error in processing commit offsets
URL: https://github.com/apache/apex-malhar/pull/644#issuecomment-315426425
   test this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Kafka input operator stops reading tuples when there is a UNKNOWN_MEMBER_ID error during
committed offset processing
> --------------------------------------------------------------------------------------------------------------------
>                 Key: APEXMALHAR-2518
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2518
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: Pramod Immaneni
>            Assignee: Pramod Immaneni
> Kafka 0.9 operator stores offsets that are completely processed and no longer needed
(committed offsets) back in kafka. It does so by making a kafka API call. If the response
from kafka server to this call comes back with an UNKNOWN_MEMBER_ID error, it results in the
kafka consumer state changing to needing partition re-assignment and no further messages are
returned by the consumer. There are a couple of other errors that result in the same state
including when rebalance is in progress.
> What exactly caused this error is not known but the following is the likely reason due
to the conditions surrounding the application. When the operator has temporarily stalled due
to back-pressure exerted by the slow downstream, it will eventually stall the operator kafka
consumer thread that is reading messages from kafka. This will result in the thread not making
any kafka consumer API calls and it will result in no heartbeats being sent to kafka server.
This can cause the server to evict the consumer after a timeout period. This could have been
the cause for the UNKNOWN_MEMBER_ID error.

This message was sent by Atlassian JIRA

View raw message