flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Morin <morin.david....@gmail.com>
Subject Re: kafka: how to stop consumption temporarily
Date Mon, 06 Jan 2020 09:35:37 GMT

Thanks for your replies.
Yes Terry. You are right. I can try to create a custom source. 
But perhaps, according to my use case, I figured out I can use a technical field in my data.
This is a timestamp and I think I just have to ignore late events with watermarks or later
in the pipeline according to metadata stored in the Flink state. I test it now...


On 2020/01/03 15:44:08, Chesnay Schepler <chesnay@apache.org> wrote: 
> Are you asking how to detect from within the job whether the dump is 
> complete, or how to combine these 2 jobs?
> If you had a way to notice whether the dump is complete, then I would 
> suggest to create a custom source that wraps 2 kafka sources, and switch 
> between them at will based on your conditions.
> On 03/01/2020 03:53, Terry Wang wrote:
> > Hi,
> >
> > I’d like to share my opinion here. It seems that you need adjust the Kafka consumer
to have communication each other. When your begin the dump process, you need to notify another
CDC-topic consumer to wait idle.
> >
> >
> > Best,
> > Terry Wang
> >
> >
> >
> >> 2020年1月2日 16:49,David Morin <morin.david.bzh@gmail.com> 写道:
> >>
> >> Hi,
> >>
> >> Is there a way to stop temporarily to consume one kafka source in streaming
mode ?
> >> Use case: I have to consume 2 topics but in fact one of them is more prioritized.
> >> One of this topic is dedicated to ingest data from db (change data capture)
and one of them is dedicated to make a synchronization (a dump i.e. a SELECT ... from db).
At the moment the last one is performed by one Flink job and we start this one after stop
the previous one (CDC) manually
> >> I want to merge these 2 modes and automatically stop consumption of the topic
dedicated to the CDC mode when a dump is done.
> >> How to handle that with Flink in a streaming way ? backpressure ? ...
> >> Thx in advance for your insights
> >>
> >> David
> >

View raw message