flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [MultipleProgramsTestBase][Cluster vs. Collection mode] Inconsistent Behavior
Date Wed, 04 Mar 2015 19:29:31 GMT
My gut feeling would be that the IterationRuntimeContext should be the same
across all iterations.

It only needs to support returning a different superstep number in each
superstep.

On Wed, Mar 4, 2015 at 7:13 PM, Vasiliki Kalavri <vasilikikalavri@gmail.com>
wrote:

> Hi,
>
> I have found the issue, but will need some help to resolve it :)
>
> In CollectionExecutor, if the superstep number is > 0 (I guess this means
> iteration?)
> a new IterationRuntimeUDFContext is created for every unary and binary
> operator, being assigned this superstep number in its constructor.
>
> However, in VertexCentricIteration, MessagingFunction and
> VertexUpdateFunction are assigned a unique IterationRuntimeContext in the
> open method of their wrappers, like this:
>
> public void open(Configuration parameters) throws Exception {
> if (getIterationRuntimeContext().getSuperstepNumber() == 1) {
> this.vertexUpdateFunction.init(getIterationRuntimeContext());
> }
> }
>
> There is no comment, but I suppose the above was written having something
> in mind that I'm not aware of...
> Is the IterationRuntimeContext supposed to be unique and the
> CollectionExecutor has to make sure to update the superstep number
> accordingly (like it does with the aggregators) or shall we assign the new
> context in every superstep?
>
> Thanks!
>
> -Vasia.
>
> On 4 March 2015 at 17:46, Vasiliki Kalavri <vasilikikalavri@gmail.com>
> wrote:
>
> > Hi Andra,
> >
> > judging from the output, it seems that all 3 supersteps are executed in
> > the second case as well,
> > but getSuperstepNumber() is returning the wrong superstep number.
> > I confirmed that this is also the case
> > in VertexCentricConnectedComponentsITCase
> > and SpargelConnectedComponentsITCase, i.e. the superstep number is wrong,
> > but the results produced are correct.
> > I'll try to find out what's wrong.
> >
> > -V.
> >
> > On 4 March 2015 at 16:31, Andra Lungu <lungu.andra@gmail.com> wrote:
> >
> >> Hello,
> >>
> >> I have implemented a Bulk Synchronous Version of Triangle Count. The
> code
> >> can be found here:
> >> https://github.com/andralungu/gelly-partitioning/tree/triangles
> >>
> >> In this algorithm, the messages sent differ as the superstep differs. In
> >> order to distinguish between superstep numbers, I used the
> >> getSuperstepNumber() function.
> >>
> >> In order to test the overall implementation, I have extended
> >> MultipleProgramsTestBase... nothing unusual until here. The problem is
> >> that
> >> in CLUSTER mode, the test passes and the result is the one expected
> >> because
> >> the superstep number changes, as can be seen below:
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Messenger]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Update]Step 2
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Messenger]Step 3
> >> [Update]Step 3
> >> [Update]Step 3
> >> [Update]Step 3
> >>
> >> For COLLECTION, the superstep number remains 1, and the result is
> >> obviously
> >> not the one I expected.
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Messenger]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >> [Update]Step 1
> >>
> >> Does anyone have an idea what could have triggered this behaviour?
> >>
> >> Thanks in advance!
> >> Andra
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message