flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabor Gevay (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5888) ForwardedFields annotation is not generating optimised execution plan in example KMeans job
Date Wed, 22 Feb 2017 17:25:44 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878779#comment-15878779
] 

Gabor Gevay commented on FLINK-5888:
------------------------------------

Just a wild guess: Is it possible that the shuffle actually ended up between the two maps,
and it is somehow not visible on the graph? I'm thinking this because I don't see how would
the annotation make the shuffle disappear (but it's possible that I'm misunderstanding something).

> ForwardedFields annotation is not generating optimised execution plan in example KMeans
job
> -------------------------------------------------------------------------------------------
>
>                 Key: FLINK-5888
>                 URL: https://issues.apache.org/jira/browse/FLINK-5888
>             Project: Flink
>          Issue Type: Bug
>          Components: DataSet API, Examples, Java API
>    Affects Versions: 1.1.3
>            Reporter: Ziyad Muhammed Mohiyudheen
>
> Flink KMeans java example [1] shows the usage of ForwardedFields function annotation.
How ever, the example job was taking more time than expected on medium sized data itself.
By merely removing the function annotation from the example code (with out any other change),
a better execution plan and run time was obtained. The execution plan shows that no combiner
is used and the two Map tasks are not chained when ForwardedFields is enabled. The experiment
is documented in [2]
> [1] https://github.com/apache/flink/blob/master/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java
> [2] https://drive.google.com/open?id=0B0IlZv0uHBuvVEZ5ZmNpN19jVVU



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message