Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6D64F10B3F for ; Wed, 18 Feb 2015 21:17:13 +0000 (UTC) Received: (qmail 63711 invoked by uid 500); 18 Feb 2015 21:17:13 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 63642 invoked by uid 500); 18 Feb 2015 21:17:13 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 63630 invoked by uid 99); 18 Feb 2015 21:17:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Feb 2015 21:17:13 +0000 X-ASF-Spam-Status: No, hits=2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vasilikikalavri@gmail.com designates 209.85.217.170 as permitted sender) Received: from [209.85.217.170] (HELO mail-lb0-f170.google.com) (209.85.217.170) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Feb 2015 21:16:47 +0000 Received: by lbvn10 with SMTP id n10so3835994lbv.4 for ; Wed, 18 Feb 2015 13:16:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=zuIIAtjS+2oJ6Ga6/dW83F2pNhovvJsnKws/wbLCRio=; b=nqCC9k4nZqw+ye5QXyh3pcIh37PN9Mutn/l3GeIgzyBGqRwPiU3H0kas1Aw5MCUAzP H04Pd46DOzsR+zLuUvMlINJ51cedSejcgsiFZ/+sc6c1XB3ph4X2ov3gGA9farmj9kbT r7SkX987bEH/dQtB+IPpQrJSntIXUf9sJ+As9lPmZZwZWYHJ2Psh8GRxT9agq7a+QhNz kWiXyjEkSCL7PfIVmJziT4KthQf42yXp4E+Q0Isb5TGM61RotZCd15PpXhpNeFZvKzRy 5SwbpuXHD/qs+trzEsTn9fbdvZn6cwAquYwgnFUwUY+k8A8u8Oy5QfAp3ygHN1MGpEox tH2w== MIME-Version: 1.0 X-Received: by 10.112.12.196 with SMTP id a4mr1356524lbc.8.1424294206187; Wed, 18 Feb 2015 13:16:46 -0800 (PST) Received: by 10.114.172.164 with HTTP; Wed, 18 Feb 2015 13:16:46 -0800 (PST) In-Reply-To: <1424289206938-745.post@n4.nabble.com> References: <1424282638421-743.post@n4.nabble.com> <1424289206938-745.post@n4.nabble.com> Date: Wed, 18 Feb 2015 22:16:46 +0100 Message-ID: Subject: Re: Using Spargel's FilterOnVerices gets stuck. From: Vasiliki Kalavri To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a11c3aa98372f83050f635635 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c3aa98372f83050f635635 Content-Type: text/plain; charset=UTF-8 Hi Hung, I am under the impression that circular dependencies like the one you are describing are not allowed in the Flink execution graph. I would actually expect something like this to cause an error. Maybe someone else can elaborate on that? In any case, the proper way to write iterative programs in Flink is by using the dedicated iteration operators. As far as I understand, you want to run an iterative algorithm on a graph, where you feed the result of one iteration to the next. Is this the case? For the moment, if you want to use a Graph API, you can either use Gelly's runVertexCentricIteration or Spargel. If your algorithm requires more flexibility, then you can build an arbitrary dataflow inside an iteration, using Flink's iteration operators. Take a look at [1] for a description of how to use those and let us know if you have any doubts. Cheers, V. [1]: http://flink.apache.org/docs/0.8/programming_guide.html#iteration-operators On 18 February 2015 at 20:53, HungChang wrote: > Thank you for your reply. > > The dataset: > The 1MB dataset is 38831 nodes and 99565 edges which doesn't get stuck. > The 30MB dataset is 1,134,890 nodes and 2,987,624 edges which gets stuck. > > Our code works like the following logic: > > do{ > > filteredGraph = graph.run(algorithm); > > // Get sub-graph for next iteration, where the code gets stuck with 30MB > dataset > filteredGraph = filteredGraph.filterOnVertices(new FilterVertex()); > > }(while filteredGraph.hasNode()) > > > filter function is as follow > > ------------------------------------------------------------------------------------------------------- > public static final class FilterVertex implements > FilterFunction Integer, Integer>> { > > @Override > public boolean filter(Tuple4 Integer> value) > throws Exception { > return value.f0 == -1; > } > } > > > > -- > View this message in context: > http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Using-Spargel-s-FilterOnVerices-gets-stuck-tp743p745.html > Sent from the Apache Flink (Incubator) User Mailing List archive. mailing > list archive at Nabble.com. > --001a11c3aa98372f83050f635635 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Hung,

I am under the impress= ion that circular dependencies like the one you are describing are not allo= wed in the Flink execution graph. I would actually expect something like th= is to cause an error.=C2=A0

Maybe someone else can elaborate on that?

In any ca= se, the proper way to write iterative programs in Flink is by using the ded= icated iteration operators.=C2=A0
As fa= r as I understand, you want to run an iterative algorithm on a graph, where= you feed the result of one iteration to the next.
Is this the case?

For the moment, if you want to use a Graph API, y= ou can either use Gelly's runVertexCentricIteration or Spargel.
If your a= lgorithm requires more flexibility, then you can build an arbitrary dataflo= w inside an iteration, using Flink's iteration operators. Take a look a= t [1] for a description of how to use those and let us know if you have any= doubts.

Cheers,
V.

[1]:=C2=A0http://flink.apache.org/docs/0.8/programming_guide.html#it= eration-operators

On 18 February 2015 at 20:53, HungChang = <unicorn.= banachi@gmail.com> wrote:
T= hank you for your reply.

The dataset:
The 1MB dataset is 38831 nodes and 99565 edges which doesn't get stuck.=
The 30MB dataset is 1,134,890 nodes and 2,987,624 edges which gets stuck.
Our code works like the following logic:

do{

filteredGraph =3D graph.run(algorithm);

// Get sub-graph for next iteration, where the code gets stuck with 30MB dataset
filteredGraph =3D filteredGraph.filterOnVertices(new FilterVertex());

}(while filteredGraph.hasNode())


filter function is as follow
---------------------------------------------------------------------------= ----------------------------
public static final class FilterVertex implements
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 FilterFunction<Tuple4&lt;Integer, Integer, Integer, Integ= er>> {

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 @Override
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 public boolean filt= er(Tuple4<Integer, Integer, Integer, Integer> value)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 throws Exception {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 return value.f0 =3D=3D -1;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }



--
View this message in context: http://apache-flink-incubator= -user-mailing-list-archive.2336050.n4.nabble.com/Using-Spargel-s-FilterOnVe= rices-gets-stuck-tp743p745.html
Sent from the Apache Flink (Incubat= or) User Mailing List archive. mailing list archive at Nabble.com.

--001a11c3aa98372f83050f635635--