Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 05CAD18F47 for ; Mon, 20 Jul 2015 13:43:36 +0000 (UTC) Received: (qmail 33480 invoked by uid 500); 20 Jul 2015 13:43:35 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 33408 invoked by uid 500); 20 Jul 2015 13:43:35 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 33398 invoked by uid 99); 20 Jul 2015 13:43:35 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Jul 2015 13:43:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 445EC189AC8 for ; Mon, 20 Jul 2015 13:43:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.9 X-Spam-Level: *** X-Spam-Status: No, score=3.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id JkagkcK3Y7wU for ; Mon, 20 Jul 2015 13:43:26 +0000 (UTC) Received: from mail-wg0-f44.google.com (mail-wg0-f44.google.com [74.125.82.44]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 26A7D2142E for ; Mon, 20 Jul 2015 13:43:26 +0000 (UTC) Received: by wgav7 with SMTP id v7so65503243wga.2 for ; Mon, 20 Jul 2015 06:43:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=y5DQAjgbQ0mAQlzNCLkq69ITZ6cuNPuPY+w8c1j34No=; b=vfLZyE6D2/9aXQ7YqCQcvxkHgo8+UJCZstwVRZnwZ27LnvwCWpjnVerus9rW/wO+PS CB6nl6HXaL6dYjwN5Trs5RNhJe/02f1ncX2yuwahwA0Yu6+W+UEOiIbbkXereSClJdw8 6QpOUm5fzvYayjS/zWr086i8JusdeF6LRm22QQbHdVy3/fDplHA1y89GfUCyBEJp9ycD xJKcXlnLONcKG8Mv/43CkBgC4/vOaV+fa1WueKdA7FZoWRGS7wY3ILO/O3TyBG2QWFFv 2faJvZAgX8+wz2ZbHmtrRuEJdgeSEUf4VsrpCBks5xap6JzIl+2uC5JQfNfTM+Dt8irE EVcw== MIME-Version: 1.0 X-Received: by 10.180.8.234 with SMTP id u10mr17639286wia.4.1437399805871; Mon, 20 Jul 2015 06:43:25 -0700 (PDT) Received: by 10.194.63.177 with HTTP; Mon, 20 Jul 2015 06:43:25 -0700 (PDT) In-Reply-To: References: Date: Mon, 20 Jul 2015 15:43:25 +0200 Message-ID: Subject: Re: Too few memory segments provided exception From: Shivani Ghatge To: user@flink.apache.org Content-Type: multipart/alternative; boundary=f46d04182704d44daf051b4eb811 --f46d04182704d44daf051b4eb811 Content-Type: text/plain; charset=UTF-8 Hello Vasia, As I had mentioned before, I need a BloomFilter as well as a HashSet for the approximation to work. In the exact solution I am getting two HashSets and comparing them. In approximate version, if we get two BloomFilters then we have no way to compare the neighborhood sets. I thought we agreed that the BloomFilters are to be sent as messages to the vertices? The exact version is passing all the tests. On removing the final GroupReduce the program is working but I need it to add the Partial Adamic Adar edges weights. On Mon, Jul 20, 2015 at 3:15 PM, Vasiliki Kalavri wrote: > Hi Shivani, > > why are you using a vertex-centric iteration to compute the approximate > Adamic-Adar? > It's not an iterative computation :) > > In fact, it should be as complex (in terms of operators) as the exact > Adamic-Adar, only more efficient because of the different neighborhood > representation. Are you having the same problem with the exact computation? > > Cheers, > Vasia. > > On 20 July 2015 at 14:41, Maximilian Michels wrote: > >> Hi Shivani, >> >> The issue is that by the time the Hash Join is executed, the >> MutableHashTable cannot allocate enough memory segments. That means that >> your other operators are occupying them. It is fine that this also occurs >> on Travis because the workers there have limited memory as well. >> >> Till suggested to change the memory fraction through the >> ExuectionEnvironment. Can you try that? >> >> Cheers, >> Max >> >> On Mon, Jul 20, 2015 at 2:23 PM, Shivani Ghatge >> wrote: >> >>> Hello Maximilian, >>> >>> Thanks for the suggestion. I will use it to check the program. But when >>> I am creating a PR for the same implementation with a Test, I am getting >>> the same error even on Travis build. So for that what would be the >>> solution? >>> >>> Here is my PR https://github.com/apache/flink/pull/923 >>> And here is the Travis build status >>> https://travis-ci.org/apache/flink/builds/71695078 >>> >>> Also on the IDE it is working fine in Collection execution mode. >>> >>> Thanks and Regards, >>> Shivani >>> >>> On Mon, Jul 20, 2015 at 2:14 PM, Maximilian Michels >>> wrote: >>> >>>> Hi Shivani, >>>> >>>> Flink doesn't have enough memory to perform a hash join. You need to >>>> provide Flink with more memory. You can either increase the >>>> "taskmanager.heap.mb" config variable or set "taskmanager.memory.fraction" >>>> to some value greater than 0.7 and smaller then 1.0. The first config >>>> variable allocates more overall memory for Flink; the latter changes the >>>> ratio between Flink managed memory (e.g. for hash join) and user memory >>>> (for you functions and Gelly's code). >>>> >>>> If you run this inside an IDE, the memory is configured automatically >>>> and you don't have control over that at the moment. You could, however, >>>> start a local cluster (./bin/start-local) after you adjusted your >>>> flink-conf.yaml and run your programs against that configured cluster. You >>>> can do that either through your IDE using a RemoteEnvironment or by >>>> submitting the packaged JAR to the local cluster using the command-line >>>> tool (./bin/flink). >>>> >>>> Hope that helps. >>>> >>>> Cheers, >>>> Max >>>> >>>> On Mon, Jul 20, 2015 at 2:04 PM, Shivani Ghatge >>>> wrote: >>>> >>>>> Hello, >>>>> I am working on a problem which implements Adamic Adar Algorithm >>>>> using Gelly. >>>>> I am running into this exception for all the Joins (including the one >>>>> that are part of the reduceOnNeighbors function) >>>>> >>>>> Too few memory segments provided. Hash Join needs at least 33 memory >>>>> segments. >>>>> >>>>> >>>>> The problem persists even when I comment out some of the joins. >>>>> >>>>> Even after using edg = edg.join(graph.getEdges(), >>>>> JoinOperatorBase.JoinHint.BROADCAST_HASH_SECOND).where(0,1).equalTo(0,1).with(new >>>>> JoinEdge()); >>>>> >>>>> as suggested by @AndraLungu the problem persists. >>>>> >>>>> The code is >>>>> >>>>> >>>>> DataSet> degrees = graph.getDegrees(); >>>>> >>>>> //get neighbors of each vertex in the HashSet for it's value >>>>> computedNeighbors = graph.reduceOnNeighbors(new >>>>> GatherNeighbors(), EdgeDirection.ALL); >>>>> >>>>> //get vertices with updated values for the final Graph which >>>>> will be used to get Adamic Edges >>>>> Vertices = computedNeighbors.join(degrees, >>>>> JoinOperatorBase.JoinHint.BROADCAST_HASH_FIRST).where(0).equalTo(0).with(new >>>>> JoinNeighborDegrees()); >>>>> >>>>> Graph, List>>>> Long, Double>>>, Double> updatedGraph = >>>>> Graph.fromDataSet(Vertices, edges, env); >>>>> >>>>> //configure Vertex Centric Iteration >>>>> VertexCentricConfiguration parameters = new >>>>> VertexCentricConfiguration(); >>>>> >>>>> parameters.setName("Find Adamic Adar Edge Weights"); >>>>> >>>>> parameters.setDirection(EdgeDirection.ALL); >>>>> >>>>> //run Vertex Centric Iteration to get the Adamic Adar Edges >>>>> into the vertex Value >>>>> updatedGraph = updatedGraph.runVertexCentricIteration(new >>>>> GetAdamicAdarEdges(), new NeighborsMessenger(), 1, parameters); >>>>> >>>>> //Extract Vertices of the updated graph >>>>> DataSet, >>>>> List>>>> vertices = updatedGraph.getVertices(); >>>>> >>>>> //Extract the list of Edges from the vertex values >>>>> DataSet> edg = vertices.flatMap(new >>>>> GetAdamicList()); >>>>> >>>>> //Partial weights for the edges are added >>>>> edg = edg.groupBy(0,1).reduce(new AdamGroup()); >>>>> >>>>> //Graph is updated with the Adamic Adar Edges >>>>> edg = edg.join(graph.getEdges(), >>>>> JoinOperatorBase.JoinHint.BROADCAST_HASH_SECOND).where(0,1).equalTo(0,1).with(new >>>>> JoinEdge()); >>>>> >>>>> Any idea how I could tackle this Exception? >>>>> >>>> >>>> >>> >> > --f46d04182704d44daf051b4eb811 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello Vasia,

As I had men= tioned before, I need a BloomFilter as well as a HashSet for the approximat= ion to work. In the exact solution I am getting two HashSets and comparing = them. In approximate version, if we get two BloomFilters then we have no wa= y to compare the neighborhood sets.

I thought we agreed that t= he BloomFilters are to be sent as messages to the vertices?

Th= e exact version is passing all the tests.

On removing the fina= l GroupReduce the program is working but I need it to add the Partial Adami= c Adar edges weights.

On Mon, Jul 20, 2015 at 3:15 PM, Vasiliki Kalavri <vasilikikalavri@gmail.com> wrote:
Hi Shivani,

why are you using a ver= tex-centric iteration to compute the approximate Adamic-Adar?
It's not an iterative computation :)=C2=A0

In fact, it should be= as complex (in terms of operators) as the exact Adamic-Adar, only more eff= icient because of the different neighborhood representation. Are you having= the same problem with the exact computation?

Cheers,
V= asia.

On 20 July 2015 at 14:41, Maximilia= n Michels <mxm@apache.org> wrote:
Hi Shivani,

The issue is that by the time the Hash Join is execut= ed, the MutableHashTable cannot allocate enough memory segments. That means= that your other operators are occupying them. It is fine that this also oc= curs on Travis because the workers there have limited memory as well.
Till suggested to change the memory frac= tion through the ExuectionEnvironment. Can you try that?

Cheers,
Max
<= /div>

On= Mon, Jul 20, 2015 at 2:23 PM, Shivani Ghatge <shghatge@gmail.com>= wrote:
Hello Maximilian,

Thanks for the suggesti= on. I will use it to check the program. But when I am creating a PR for the= same implementation with a Test, I am getting the same error even on Travi= s build. So for that what would be the solution?

Here is my P= R ht= tps://github.com/apache/flink/pull/923
And here is the Travis = build status https://travis-ci.org/apache/flink/builds/71695078
Also on the IDE it is working fine in Collection execution mode= .

Thanks and Regards,
Shivani

On Mon, Jul 20, 2015 at = 2:14 PM, Maximilian Michels <mxm@apache.org> wrote:
<= div>
Hi Shiva= ni,

Flink doesn't have enough memory to perform a hash join. You= need to provide Flink with more memory. You can either increase the "= taskmanager.heap.mb" config variable or set "taskmanager.memory.f= raction" to some value greater than 0.7 and smaller then 1.0. The firs= t config variable allocates more overall memory for Flink; the latter chang= es the ratio between Flink managed memory (e.g. for hash join) and user mem= ory (for you functions and Gelly's code).

If you run = this inside an IDE, the memory is configured automatically and you don'= t have control over that at the moment. You could, however, start a local c= luster (./bin/start-local) after you adjusted your flink-conf.yaml and run = your programs against that configured cluster. You can do that either throu= gh your IDE using a RemoteEnvironment or by submitting the packaged JAR to = the local cluster using the command-line tool (./bin/flink).

<= div>Hope that helps.

Cheers,
Max
=

On Mon,= Jul 20, 2015 at 2:04 PM, Shivani Ghatge <shghatge@gmail.com> wrote:
Hello,
=C2=A0I am working on a problem which implements Adamic Ad= ar Algorithm using Gelly.
I am running into this exception for all= the Joins (including the one that are part of the reduceOnNeighbors functi= on)

Too few memory segments provided. Hash Join needs at least 33 me= mory segments.


The problem persists even when I comme= nt out some of the joins.

Even after using edg =3D edg.jo= in(graph.getEdges(), JoinOperatorBase.JoinHint.BROADCAST_HASH_SECOND).where= (0,1).equalTo(0,1).with(new JoinEdge());

as suggested by = @AndraLungu the problem persists.

The code is

DataSet<Tuple2<Long, Long>> degrees =3D graph.getDegrees();
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 //get neighbors of each vertex i= n the HashSet for it's value
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 c= omputedNeighbors =3D graph.reduceOnNeighbors(new GatherNeighbors(), EdgeDir= ection.ALL);
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2= =A0 =C2=A0=C2=A0=C2=A0 //get vertices with updated values for the final Gra= ph which will be used to get Adamic Edges
=C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 Vertices =3D computedNeighbors.join(degrees, JoinOperatorBase.Joi= nHint.BROADCAST_HASH_FIRST).where(0).equalTo(0).with(new JoinNeighborDegree= s());

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 Graph<Long, Tuple3<= ;Double, HashSet<Long>, List<Tuple3<Long, Long, Double>>&= gt;, Double> updatedGraph =3D
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 Graph.fromDataSet(Vertices, edges, en= v);
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 //configure Vertex Centric Iteration
=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 VertexCentricConfiguration parameters =3D new VertexCentric= Configuration();

=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 parameters.se= tName("Find Adamic Adar Edge Weights");

=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 parameters.setDirection(EdgeDirection.ALL);
=C2=A0= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 /= /run Vertex Centric Iteration to get the Adamic Adar Edges into the vertex = Value
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 updatedGraph =3D updatedGra= ph.runVertexCentricIteration(new GetAdamicAdarEdges<Long>(), new Neig= hborsMessenger<Long>(), 1, parameters);
=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 //Extract Vertices o= f the updated graph
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 DataSet<Ver= tex<Long, Tuple3<Double, HashSet<Long>, List<Tuple3<Long,= Long, Double>>>>> vertices =3D updatedGraph.getVertices();<= br>=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 //Extract the list of Edges from the vertex values
=C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 DataSet<Tuple3<Long, Long, Double>>= ; edg =3D vertices.flatMap(new GetAdamicList());
=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 //Partial weights= for the edges are added
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 edg =3D e= dg.groupBy(0,1).reduce(new AdamGroup());

=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 //Graph is updated with the Adamic Adar Edges
=C2=A0=C2=A0= =C2=A0 =C2=A0=C2=A0=C2=A0 edg =3D edg.join(graph.getEdges(), JoinOperatorBa= se.JoinHint.BROADCAST_HASH_SECOND).where(0,1).equalTo(0,1).with(new JoinEdg= e());

Any idea how I could tackle this Exception?





--f46d04182704d44daf051b4eb811--