flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fabian Hueske (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-1656) Fix ForwardedField documentation for operators with iterator input
Date Fri, 06 Mar 2015 10:29:38 GMT
Fabian Hueske created FLINK-1656:
------------------------------------

             Summary: Fix ForwardedField documentation for operators with iterator input
                 Key: FLINK-1656
                 URL: https://issues.apache.org/jira/browse/FLINK-1656
             Project: Flink
          Issue Type: Bug
          Components: Documentation
    Affects Versions: 0.9
            Reporter: Fabian Hueske
            Assignee: Fabian Hueske
            Priority: Critical


The documentation of ForwardedFields is incomplete for operators with iterator inputs (GroupReduce,
CoGroup). 
This should be fixed ASAP, because it can lead to incorrect program execution.

The conditions for forwarded fields on operators with iterator input are:

1) forwarded fields must be emitted in the order in which they are received through the iterator
2) all forwarded fields of a record must stick together, i.e., if your function builds record
from field 0 of the 1st, 3rd, 5th, ... and field 1 of the 2nd, 4th, ... record coming through
the iterator, these are not valid forwarded fields.
3) it is OK to completely filter out records coming through the iterator.

The reason for these conditions is that the optimizer uses forwarded fields to reason about
physical data properties such as order and grouping. Mixing up the order of records or emitting
records which are composed from different input records, might destroy a (secondary) order
or grouping.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message