flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From senorcarbone <...@git.apache.org>
Subject [GitHub] flink issue #1668: [FLINK-3257] Add Exactly-Once Processing Guarantees for I...
Date Tue, 13 Dec 2016 09:16:12 GMT
Github user senorcarbone commented on the issue:

    These are some good points @StephanEwen, thanks for checking it. 
    How about the following, regarding each issue:
    - `Concurrent Checkpoints`: Looks like an improvement but I can sure do it in this PR
if  it is a crucial one. Can you elaborate a bit more or point me out to other concurrent
checkpointing operator state examples to get an idea of how you want to do it?
    - `Reconfiguration` : Sounds interesting...but I am not really aware of it from the devlist.
If it is simple enough I could add support for it here. Otherwise I would suggest we address
this in a seperate JIRA and PR as an improvement. Is there a design document on how we plan
to achieve reconfiguration and repartitioning for operator state specifically somewhere?
    - `At-most-once blocking queue` : It is obvious from my previous comments that I do not
approve this part, but that is something we already got rid of in [FLIP-15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-15+Scoped+Loops+and+Job+Termination)
already ([this](https://github.com/FouadMA/flink/commit/9adaac435bcaf3552afe564c739d4e8fd79c433b)
commit). How about we address this together with the deadlocks in FLIP-15?
    - `Deadlocks`: I like the elastic spilling channel idea to resolve deadlocks. I need time
to dig a bit more into this and make sure we solve deadlocks and not just improve. Is it ok
with you if we address that in [FLIP-15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-15+Scoped+Loops+and+Job+Termination)?
I need more time for this part, plus, we need to combine the absense of expiring queues with
a proper termination algorithm (otherwise we just solve the deadlocks and the jobs never terminate).
    What do you think?

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message