Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Wed, 15 Jul 2015 09:23:04 +0000 (UTC)
From: "Vasia Kalavri (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.12845089.1436909442000.193109.1436952184537@Atlassian.JIRA>
In-Reply-To: <JIRA.12845089.1436909442000@Atlassian.JIRA>
References: <JIRA.12845089.1436909442000@Atlassian.JIRA>
 <JIRA.12845089.1436909442913@arcas>
Subject: [jira] [Commented] (FLINK-2361) flatMap + distinct gives erroneous
 results for big data sets
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FLINK-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627773#comment-14627773 ] 

Vasia Kalavri commented on FLINK-2361:
--------------------------------------

Hey,

I have seen this exception before and I think it was when debugging FLINK-1930.
I don't think this is an operator correctness problem either.
[~andralungu], I take it vertex 657282846 actually exists in the dataset and it's not garbage, right?
As far as I remember, what seems to be happening is that the vertex dataset has not been fully generated when the message to this vertex ID is sent, i.e. some kind of blocking issue.
Can you check if this problem is still there when you generate the vertex set separately, store it to disk and get input from edges and vertex files?

> flatMap + distinct gives erroneous results for big data sets
> ------------------------------------------------------------
>
>                 Key: FLINK-2361
>                 URL: https://issues.apache.org/jira/browse/FLINK-2361
>             Project: Flink
>          Issue Type: Bug
>          Components: Gelly
>    Affects Versions: 0.10
>            Reporter: Andra Lungu
>
> When running the simple Connected Components algorithm (currently in Gelly) on the twitter follower graph, with 1, 100 or 10000 iterations, I get the following error:
> Caused by: java.lang.Exception: Target vertex '657282846' does not exist!.
> 	at org.apache.flink.graph.spargel.VertexCentricIteration$VertexUpdateUdfSimpleVV.coGroup(VertexCentricIteration.java:300)
> 	at org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDriver.run(CoGroupWithSolutionSetSecondDriver.java:220)
> 	at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:496)
> 	at org.apache.flink.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:139)
> 	at org.apache.flink.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:107)
> 	at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> 	at java.lang.Thread.run(Thread.java:722)
> Now this is very bizzare as the DataSet of vertices is produced from the DataSet of edges... Which means there cannot be a an edge with an invalid target id... The method calls flatMap to isolate the src and trg ids and distinct to ensure their uniqueness.  
> The algorithm works fine for smaller data sets... 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)