camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shashank Madhav (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CAMEL-10333) (Consumer goes into loop while processing messages) DEBUG Cannot auto-commit offsets now since the coordinator is unknown
Date Tue, 20 Sep 2016 16:49:20 GMT
Shashank Madhav created CAMEL-10333:
---------------------------------------

             Summary: (Consumer goes into loop while processing messages) DEBUG Cannot auto-commit
offsets now since the coordinator is unknown
                 Key: CAMEL-10333
                 URL: https://issues.apache.org/jira/browse/CAMEL-10333
             Project: Camel
          Issue Type: Bug
          Components: camel-kafka
    Affects Versions: 2.17.1
         Environment: Spring Framework is used for creating consumers, which processes the
messages.
            Reporter: Shashank Madhav


I have a topic which have 300 messages in the queue. When the camel is used for processing
the messages with event-based consumer, I am getting following error after all the messages
are processed.

/////error message start
[ad #0 - KafkaTopic[Audit_logs]] ConsumerCoordinator            DEBUG Cannot auto-commit offsets
now since the coordinator is unknown, will retry after backoff
[ad #0 - KafkaTopic[Audit_logs]] AbstractCoordinator            DEBUG Issuing group metadata
request to broker 0
[ad #0 - KafkaTopic[Audit_logs]] AbstractCoordinator            DEBUG Group metadata response
ClientResponse(receivedTimeMs=1473953020273, disconnected=false, request=ClientRequest(expectResponse=true,
callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@1edc39d,
request=RequestSend(header={api_key=10,api_version=0,correlation_id=9,client_id=consumer-1},
body={group_id=test3}), createdTimeMs=1473953020218, sendTimeMs=1473953020218), responseBody={error_code=0,coordinator={node_id=0,host=<host-name>,port=<port-no>}})
[ad #0 - KafkaTopic[Audit_logs]] ConsumerCoordinator            ERROR Error ILLEGAL_GENERATION
occurred while committing offsets for group test3
[ad #0 - KafkaTopic[Audit_logs]] ConsumerCoordinator            WARN  Auto offset commit failed:
Commit cannot be completed due to group rebalance
[ad #0 - KafkaTopic[Audit_logs]] ConsumerCoordinator            ERROR Error ILLEGAL_GENERATION
occurred while committing offsets for group test3
[ad #0 - KafkaTopic[Audit_logs]] ConsumerCoordinator            WARN  Auto offset commit failed:

////error message end

 Following the error, it then starts from the beginning and processes the entire queue again
causing issues. The route used for camel is as follows 

//camel-route start
        <c:endpoint id="kafkaTest" uri="kafka:${broker.server}?topic=${topics1}&amp;groupId=test3&amp;autoOffsetReset=earliest&amp;consumersCount=1&amp;sessionTimeoutMs=30000&amp;consumerRequestTimeoutMs=40000&amp;keyDeserializer=${broker.keyDeserializer}&amp;valueDeserializer=${broker.valueDeserializer}"/>
		
		<c:route id="route1">
            <c:from uri="ref:kafkaTest"/>
            <c:delay>
            <c:constant>500</c:constant>
            </c:delay>
             <c:to uri="log:input"/> 
        </c:route>
//camel-route ends

Delay is purposely added in the route to create message processing scenario. If the delay
is reduced or the sessionTimeoutMs is increased then it works fine, but this is not a feasible
solution. Without camel it works fine and messages are processed properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message