beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <>
Subject [jira] [Commented] (BEAM-1393) Update Flink Runner to Flink 1.2.0
Date Tue, 07 Feb 2017 16:34:41 GMT


Aljoscha Krettek commented on BEAM-1393:

Yes, there are definitely some changes that need to be done while moving to Flink 1.2.

1. This should be covered by BEAM-1394.
2. I think we can remove the checks and use {{SourceFunction.SourceContext}} instead.
3. Definitely true, this will get us rescalability.
4. I think it should be {{GlobalConfiguration.loadConfiguration()}}, AFAIK the dynamic properties
is only used in conjunction with Mesos code so far
5. true
6. I think we have to get a bit creative here and only call super when we know that the watermark
should advance. Casting doesn't seem wise since we won't always be guaranteed that the timer
service is in fact a {{HeapInternalTimerService}}.
7. What BROADCAST does is collect all checkpointed states into one "list" and then send out
that list to all parallel subtasks when restoring. The way we would use it is to only checkpoint
anything from the operator with subtask index 0 because we assume that the state is the same
on all parallel instances of the operator.

That last part is still a bit vague, I think we can tackle that once the other parts are figured

> Update Flink Runner to Flink 1.2.0
> ----------------------------------
>                 Key: BEAM-1393
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Aljoscha Krettek
>            Assignee: Jingsong Lee
> When we update to 1.2.0 we can use the new internal Timer API that is available to Flink
operators: {{InternalTimerService}} and also use broadcast state to store side-input data.

This message was sent by Atlassian JIRA

View raw message