flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tzulitai <...@git.apache.org>
Subject [GitHub] flink pull request #2414: [FLINK-4341] Let idle consumer subtasks emit max v...
Date Wed, 24 Aug 2016 17:12:20 GMT
GitHub user tzulitai opened a pull request:

    https://github.com/apache/flink/pull/2414

    [FLINK-4341] Let idle consumer subtasks emit max value watermarks and fail on resharding

    This is a short-term fix, until the min-watermark service for the JobManager described
in the JIRA discussion is available.
    
    The way this fix works is that we let idle subtasks that initially don't get assigned
shards emit a `Long.MAX_VALUE` watermark. Also, we _only fail hard if an idle subtask_ is
assigned new shards when resharding happens, to avoid messing up the watermarks. So, if all
subtasks are not initially idle on startup (i.e., when total number of shards > consumer
parallelism), the Kinesis consumer can still transparently handle resharding like before without
failing.
    
    I've tested exactly-once with our manual tests (with and w/o resharding) and the fix works
nicely, still retaining exactly-once guarantee despite non-transparency. However, I'm a bit
unsure on how to test if the unbounded state with window operators is also fixed with this
change, so we're still yet to clarify this.
    
    R: @rmetzger and @aljoscha for review. Thanks in advance!

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tzulitai/flink FLINK-4341

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/2414.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2414
    
----
commit bc8e50d99be745300f7418c58e9d30abc5469ba3
Author: Gordon Tai <tzulitai@gmail.com>
Date:   2016-08-24T08:38:06Z

    [FLINK-4341] Let idle consumer subtasks emit max value watermarks and fail on resharding
    
    This no longer allows the Kinesis consumer to transparently handle resharding.
    This is a short-term workaround until we have a min-watermark notification service available
in the JobManager.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message