flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasiliki Kalavri <vasilikikala...@gmail.com>
Subject Re: [MultipleProgramsTestBase][Cluster vs. Collection mode] Inconsistent Behavior
Date Wed, 04 Mar 2015 22:09:38 GMT
Thanks!

I created an issue [1] and will try to look into this during the weekend
(unless someone gets to it faster :))

Cheers,
V.

[1]: http://issues.apache.org/jira/browse/FLINK-1652

On 4 March 2015 at 20:29, Stephan Ewen <sewen@apache.org> wrote:

> My gut feeling would be that the IterationRuntimeContext should be the same
> across all iterations.
>
> It only needs to support returning a different superstep number in each
> superstep.
>
> On Wed, Mar 4, 2015 at 7:13 PM, Vasiliki Kalavri <
> vasilikikalavri@gmail.com>
> wrote:
>
> > Hi,
> >
> > I have found the issue, but will need some help to resolve it :)
> >
> > In CollectionExecutor, if the superstep number is > 0 (I guess this means
> > iteration?)
> > a new IterationRuntimeUDFContext is created for every unary and binary
> > operator, being assigned this superstep number in its constructor.
> >
> > However, in VertexCentricIteration, MessagingFunction and
> > VertexUpdateFunction are assigned a unique IterationRuntimeContext in the
> > open method of their wrappers, like this:
> >
> > public void open(Configuration parameters) throws Exception {
> > if (getIterationRuntimeContext().getSuperstepNumber() == 1) {
> > this.vertexUpdateFunction.init(getIterationRuntimeContext());
> > }
> > }
> >
> > There is no comment, but I suppose the above was written having something
> > in mind that I'm not aware of...
> > Is the IterationRuntimeContext supposed to be unique and the
> > CollectionExecutor has to make sure to update the superstep number
> > accordingly (like it does with the aggregators) or shall we assign the
> new
> > context in every superstep?
> >
> > Thanks!
> >
> > -Vasia.
> >
> > On 4 March 2015 at 17:46, Vasiliki Kalavri <vasilikikalavri@gmail.com>
> > wrote:
> >
> > > Hi Andra,
> > >
> > > judging from the output, it seems that all 3 supersteps are executed in
> > > the second case as well,
> > > but getSuperstepNumber() is returning the wrong superstep number.
> > > I confirmed that this is also the case
> > > in VertexCentricConnectedComponentsITCase
> > > and SpargelConnectedComponentsITCase, i.e. the superstep number is
> wrong,
> > > but the results produced are correct.
> > > I'll try to find out what's wrong.
> > >
> > > -V.
> > >
> > > On 4 March 2015 at 16:31, Andra Lungu <lungu.andra@gmail.com> wrote:
> > >
> > >> Hello,
> > >>
> > >> I have implemented a Bulk Synchronous Version of Triangle Count. The
> > code
> > >> can be found here:
> > >> https://github.com/andralungu/gelly-partitioning/tree/triangles
> > >>
> > >> In this algorithm, the messages sent differ as the superstep differs.
> In
> > >> order to distinguish between superstep numbers, I used the
> > >> getSuperstepNumber() function.
> > >>
> > >> In order to test the overall implementation, I have extended
> > >> MultipleProgramsTestBase... nothing unusual until here. The problem is
> > >> that
> > >> in CLUSTER mode, the test passes and the result is the one expected
> > >> because
> > >> the superstep number changes, as can be seen below:
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Messenger]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Update]Step 2
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Messenger]Step 3
> > >> [Update]Step 3
> > >> [Update]Step 3
> > >> [Update]Step 3
> > >>
> > >> For COLLECTION, the superstep number remains 1, and the result is
> > >> obviously
> > >> not the one I expected.
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Messenger]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >> [Update]Step 1
> > >>
> > >> Does anyone have an idea what could have triggered this behaviour?
> > >>
> > >> Thanks in advance!
> > >> Andra
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message