flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giuliano Caliari <giuliano.cali...@gmail.com>
Subject Re: Flink requesting external web service with rate limited requests
Date Tue, 28 Feb 2017 23:58:15 GMT
Hey Fabian,

One of my solutions implements the AsyncFunction but I'm still unable to
savepoint because Flink reads the backed up records, thousands of historical
records, right off the bat and when I issue a savepoint request it has to
wait for all those records to be processed which takes a couple of hours. So
I'm still getting the error when savepointing. 
Alternatively I could wait for the backed up records to be processed and
issue savepoints afterwards but there is a risk of failures and I would have
to restart the whole process. 

Another idea would be if we could commit the Kafka offset only after we get
a positive response from the external web service. There would be some
duplication in case of errors but that's acceptable. Is there any easy way
we can do this?

Cheers,



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-requesting-external-web-service-with-rate-limited-requests-tp11952p11977.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message