flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ventura Del Monte <venturadelmo...@gmail.com>
Subject Re: Broadcasted Variable not updated in a Iterative Data Set
Date Tue, 12 May 2015 22:42:00 GMT
Thank you for your clarification. At the end I figured out, it was a silly
programming error, now it just works fine. Thank you again for your

2015-05-11 22:31 GMT+02:00 Stephan Ewen <sewen@apache.org>:

> Can you share the respective part of the code, then we can have a look?
> In general, a DataSet is re-broadcasted in each iteration, if it depends
> on the IterativeDataSet.
> You have to re-grab it from the RuntimeContext in each superstep. If you
> call getBroadcastSet() in the open() method, then it will get executed in
> each superstep (open gets called at the beginning of each superstep).
> Greetings,
> Stephan
> On Mon, May 11, 2015 at 6:36 PM, Ventura Del Monte <
> venturadelmonte@gmail.com> wrote:
>> Hello,
>> I am writing a Flink application whose purpose is to train neural
>> networks. I have implemented a version of SDG based on mini batches using
>> the IterativeDataSet class. In a nutshell, my code groups the input
>> features in a set called mini batch (i have as many mini batches as the
>> parallelism degree), then it calculates a gradient over each mini batch in
>> a map tasks and then those gradients are merged in a reduce task.
>> Unfortunately it seems the trained param is not broadcasted at the end of
>> each epoch and it looks like that it starts the next epoch using the
>> initial parameter. I know it is probably a programming error which I cannot
>> see at the moment, but I was wondering if there were some guidelines for
>> broadcasting an object in the correct way.
>> Is it asking too much if you could have a look at my code, please? Thank
>> you in advance.
>> Regards,
>> Ventura

View raw message