flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Hutchison (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-6114) Type checking fails with generics, even when concrete type of field is not needed
Date Tue, 21 Mar 2017 05:17:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934109#comment-15934109
] 

Luke Hutchison edited comment on FLINK-6114 at 3/21/17 5:17 AM:
----------------------------------------------------------------

Hi Greg, the code that caused the exception was doing the following, if memory serves correctly:

{code}
    public static <K> DataSet<Tuple3<String, K, Float>> doSomething(
            DataSet<Tuple3<String, K, Float>> input) {
        return input.groupBy(0).sum(2);
    }
{code}

although I realize now that this leaves field 1 in an indeterminate state: if not all fields
of a tuple are covered in the {{groupBy}} and {{sum}} parameters, and if different input tuples
have different values for the unspecified fields, then aggregation has to either pick one
of the field 1 values, or pick none of them (e.g. set field 1 to null).

I assume that this is what caused the exception, because the typechecker was not expecting
any fields to be dropped like this -- but if so, the correct exception message would be something
along the lines of {{"sum aggregation on field 2 after grouping by field 0 leaves field 1
undefined"}} (or similar).


was (Author: lukehutch):
Hi Greg, the code that caused the exception was doing the following, if memory serves correctly:

{{code}}
    public static <K> DataSet<Tuple3<String, K, Float>> doSomething(
            DataSet<Tuple3<String, K, Float>> input) {
        return input.groupBy(0).sum(2);
    }
{{code}}

although I realize now that this leaves field 1 in an indeterminate state: if not all fields
of a tuple are covered in the {{groupBy}} and {{sum}} parameters, and if different input tuples
have different values for the unspecified fields, then aggregation has to either pick one
of the field 1 values, or pick none of them (e.g. set field 1 to null).

I assume that this is what caused the exception, because the typechecker was not expecting
any fields to be dropped like this -- but if so, the correct exception message would be something
along the lines of "sum aggregation on field 2 after grouping by field 0 leaves field 1 undefined"
(or similar).

> Type checking fails with generics, even when concrete type of field is not needed
> ---------------------------------------------------------------------------------
>
>                 Key: FLINK-6114
>                 URL: https://issues.apache.org/jira/browse/FLINK-6114
>             Project: Flink
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2.0
>            Reporter: Luke Hutchison
>
> The Flink type checker does not allow generic types to be used in any field of a tuple
when a join is being executed, even if the generic is not in a field that is involved in the
join.
> I have a type Tuple3<String, K, Float>, which contains a generic type parameter
K. I am joining using .where(0).equalTo(0). The type of field 0 is well-defined as String.
However, this gives me the following error:
> {noformat}
> Exception in thread "main" org.apache.flink.api.common.functions.InvalidTypesException:
Type of TypeVariable 'K' in 'public static org.apache.flink.api.java.DataSet mypkg.MyClass.method(params)'
could not be determined. This is most likely a type erasure problem. The type extraction currently
supports types with generic variables only in cases where all variables in the return type
can be deduced from the input type(s).
> 	at org.apache.flink.api.java.typeutils.TypeExtractor.createSubTypesInfo(TypeExtractor.java:989)
> {noformat}
> The code compiles fine, however -- the static type system is able to correctly resolve
the types in the surrounding code.
> Really only the fields that are affected by joins (or groupBy, aggregation etc.) should
be checked for concrete types in this way.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message