flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasiliki Kalavri <vasilikikala...@gmail.com>
Subject Re: Using Spargel's FilterOnVerices gets stuck.
Date Wed, 18 Feb 2015 21:16:46 GMT
Hi Hung,

I am under the impression that circular dependencies like the one you are
describing are not allowed in the Flink execution graph. I would actually
expect something like this to cause an error.

Maybe someone else can elaborate on that?

In any case, the proper way to write iterative programs in Flink is by
using the dedicated iteration operators.
As far as I understand, you want to run an iterative algorithm on a graph,
where you feed the result of one iteration to the next.
Is this the case?

For the moment, if you want to use a Graph API, you can either use Gelly's
runVertexCentricIteration or Spargel.

If your algorithm requires more flexibility, then you can build an
arbitrary dataflow inside an iteration, using Flink's iteration operators.
Take a look at [1] for a description of how to use those and let us know if
you have any doubts.

Cheers,
V.

[1]:
http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators
<https://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators>

On 18 February 2015 at 20:53, HungChang <unicorn.banachi@gmail.com> wrote:

> Thank you for your reply.
>
> The dataset:
> The 1MB dataset is 38831 nodes and 99565 edges which doesn't get stuck.
> The 30MB dataset is 1,134,890 nodes and 2,987,624 edges which gets stuck.
>
> Our code works like the following logic:
>
> do{
>
> filteredGraph = graph.run(algorithm);
>
> // Get sub-graph for next iteration, where the code gets stuck with 30MB
> dataset
> filteredGraph = filteredGraph.filterOnVertices(new FilterVertex());
>
> }(while filteredGraph.hasNode())
>
>
> filter function is as follow
>
> -------------------------------------------------------------------------------------------------------
> public static final class FilterVertex implements
>                         FilterFunction<Tuple4&lt;Integer, Integer,
> Integer, Integer>> {
>
>                 @Override
>                 public boolean filter(Tuple4<Integer, Integer, Integer,
> Integer> value)
>                                 throws Exception {
>                         return value.f0 == -1;
>                 }
>         }
>
>
>
> --
> View this message in context:
> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Using-Spargel-s-FilterOnVerices-gets-stuck-tp743p745.html
> Sent from the Apache Flink (Incubator) User Mailing List archive. mailing
> list archive at Nabble.com.
>

Mime
View raw message