flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Flink Kafka Consumer Behaviour
Date Thu, 04 Aug 2016 10:06:37 GMT
Hi!

I have not used the Kafka Offset Checker before, maybe someone who worked
with that can chime in.

Greetings,
Stephan


On Wed, Aug 3, 2016 at 4:59 PM, Janardhan Reddy <janardhan.reddy@olacabs.com
> wrote:

> I can see that offsets are stored in zookeeper and are not returned when i
> query through kafka offset checker.
>
> Can you please tell me how to monitor kafka flink consumer lag for 0.8
> flink kafka consumer.
>
> On Wed, Aug 3, 2016 at 3:29 PM, Stephan Ewen <sewen@apache.org> wrote:
>
>> Hi!
>>
>> Just check in the code. The 0.8 FlinkKafkaConsumer should always commit
>> offsets, regardless of whether checkpointing is enables. The 0.9
>> FlinkKafkaConsumer actually does not do any periodic offset committing when
>> checkpointing is disabled.
>>
>> Greetings,
>> Stephan
>>
>> On Wed, Aug 3, 2016 at 7:36 AM, Janardhan Reddy <
>> janardhan.reddy@olacabs.com> wrote:
>>
>>> Checkpointing wasn't enabled in the streaming job, but the offsets
>>> should have been committed to zookeeper.
>>>
>>> But we don't see the offsets being written to zookeeper.
>>>
>>> On Tue, Aug 2, 2016 at 7:41 PM, Till Rohrmann <trohrmann@apache.org>
>>> wrote:
>>>
>>>> Hi Janardhan,
>>>>
>>>> Flink should commit the current offsets to Zookeeper whenever a
>>>> checkpoint has been completed. In case that you disabled checkpointing,
>>>> then the offsets will be periodically committed to ZooKeeper. The default
>>>> value is 60s.
>>>>
>>>> Could it be that there wasn't yet a completed checkpoint? Which version
>>>> of Flink are you using?
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Tue, Aug 2, 2016 at 7:26 PM, Janardhan Reddy <
>>>> janardhan.reddy@olacabs.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> When the run the following command i am getting that no topic is
>>>>> available for that consumer group.  i am suing
>>>>> flink-connector-kafka-0.8_${scala.version}(2.11).
>>>>>
>>>>> ./bin/kafka-consumer-groups.sh --zookeeper <> --group <>
--describe
>>>>>
>>>>> No topic available for consumer group provided
>>>>>
>>>>>
>>>>> Does the kafka consumer commit offset to the broker always ? Do we
>>>>> need to enable checkpointing for the offsets to be committed ?
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message