flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andra Lungu <lungu.an...@gmail.com>
Subject Re: [Delta Iterations] The dirty insides(insights)
Date Fri, 20 Mar 2015 11:00:21 GMT
Hi Stephan,

What I am trying to do, among other things, is to make the vertex's
inDegree and outDegree available in the vertexUpdate and sendMessages
functions via vertex.getInDegrees().
That is almost done: https://github.com/andralungu/flink/tree/spargelExt

Only the problem is that the inDegrees are computed once, per graph before
starting a vertex centric iteration. So, what I decided to do was to store
the inDegree for each vertex within the Vertex object as an attribute. So
far so good,

DataSet<Vertex<K, VV>> newVertices =
verticesWithInDegrees.runOperation(iteration);

The vertices with in degrees are the initial vertices with the degrees set
as values.
When runOperation() gets called, precisely when

createResult()

from VertexCentricIteration gets called, the inDegrees stored are lost.
This happens because basically in each superstep a new DataSet is created.

I wanted to know how to keep the degrees there. In other words,
how/where/what are the steps for vertex value updates and how to include
the degrees there?

Thanks!
Andra



On Fri, Mar 20, 2015 at 11:28 AM, Stephan Ewen <sewen@apache.org> wrote:

> Hi Andra!
>
> I am not sure I am getting exactly what the question is. The code you
> pasted is from the Spargel API - specifically just forwarding registered
> broadcast variables.
>
> What do you mean with "the vertex values get reset" ?
>
> Stephan
>
> PS: The delta iterations are based in this paper:
> http://arxiv.org/pdf/1208.0088.pdf
>
>
> On Thu, Mar 19, 2015 at 9:25 AM, Andra Lungu <lungu.andra@gmail.com>
> wrote:
>
> > Hello,
> >
> > I've used delta iterations several times up until now, but I just
> realized
> > that I never fully understood what happens inside. And the documentation
> > only explains things from a user's perspective. Which is why I could
> really
> > use your help :).
> >
> > Here goes nothing:
> > In Gelly, Graph.java, there is a nice runVertexCentricIteration(...)
> > method, which is in fact Spargel's - that disguises a delta iteration.
> What
> > I am trying to do is to set the vertex value before running
> >
> > DataSet<Vertex<K, VV>> newVertices =
> > verticesWithInDegrees.runOperation(iteration);
> >
> > The problem is that after this runOperation, the vertex values get reset.
> > Now, when I looked in VertexCentricIteration.java's createResult().
> >
> > It's a plain delta iteration that (more or less) looks like this:
> >
> > final DeltaIteration<Vertex<VertexKey, VertexValue>, Vertex<VertexKey,
> > VertexValue>> iteration =
> >    this.initialVertices.iterateDelta(this.initialVertices,
> > this.maximumNumberOfIterations, zeroKeyPos);
> >
> > ................
> >
> > // configure coGroup update function with name and broadcast variables
> > updates = updates.name("Vertex State Updates");
> > for (Tuple2<String, DataSet<?>> e : this.bcVarsUpdate) {
> >    updates = updates.withBroadcastSet(e.f1, e.f0);
> > }
> >
> > // let the operator know that we preserve the key field
> > updates.withForwardedFieldsFirst("0").withForwardedFieldsSecond("0");
> >
> > return iteration.closeWith(updates, updates);
> >
> > The
> >
> > DataSet<?>
> >
> > in the for is the problem. Vertex values get reset. Can someone give me a
> > hint on how to propagate the vertex value throughout the iteration?
> >
> > Thanks!
> > Andra
> >
> >
> ----------------------------------------------------------------------------------------
> > P.S. Could someone please subscribe me to the @user mailing list? For
> some
> > reason, sending the mail to that recipient fails. Thanks!
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message