flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcela Charfuelan <charfuelanol...@tu-berlin.de>
Subject Re: Iterations problem in command line
Date Tue, 01 Mar 2016 10:59:18 GMT
Hi,
the iteration looks like:
DataSet<GMM> gmms = getInitialGMMDataSet(env);
		
IterativeDataSet<GMM> loop = gmms.iterate(50);

DataSet<GMM> newGMMs = features.map(new 
Estep_ExpectationMaximisation()).withBroadcastSet(loop, "gmms")
		       .reduceGroup(new 
Mstep_ExpectationMaximisation()).withBroadcastSet(loop, "gmms");
	
DataSet<GMM> finalGMMs = loop.closeWith(newGMMs)

in every iteration the gmms parameters should be updated, I have noticed 
that the first iteration is ok, but afterwards start to get wrong... at 
least in the command line (for example gmms.coeff for the three gmms 
here should not sum up more that 1)

I have put the code here in case it helps:
https://github.com/marcelach1/EmExercise

Regards,
Marcela.


On 01.03.2016 10:58, Fabian Hueske wrote:
> Yes, env.setParallelism(1) fixes the parallelism of all operators to 1
> (unless an operator overrides this setting).
> Can you identify at which position in the data flow the results start to
> diverge?
>
> Best, Fabian
>
> 2016-02-29 17:57 GMT+01:00 Marcela Charfuelan
> <charfuelanoliva@tu-berlin.de <mailto:charfuelanoliva@tu-berlin.de>>:
>
>     Thanks Fabian,
>     I am using in both default options, since I am not testing in a
>     cluster yet, just local in ubuntu, I am not specifying any parallelism.
>     just to test I set in the program env.setParallelism(1) and running
>     with -p 1 (which I guess I would not need) but I am still getting
>     the same issue.
>
>     Regards,
>     MArcela.
>
>
>     On 29.02.2016 16:44, Fabian Hueske wrote:
>
>         Hi Marcela,
>
>         do you run the algorithm in both setups with the same parallelism?
>
>         Best, Fabian
>
>         2016-02-26 16:52 GMT+01:00 Marcela Charfuelan
>         <charfuelanoliva@tu-berlin.de
>         <mailto:charfuelanoliva@tu-berlin.de>
>         <mailto:charfuelanoliva@tu-berlin.de
>         <mailto:charfuelanoliva@tu-berlin.de>>>:
>
>              Hello,
>
>              I implemented an algorithm that includes iterations (EM
>         algorithm)
>              and I am getting different results when running in eclipse
>         (Luna
>              Release (4.4.0)) and when running in the command line using
>         Flink
>              run; the program does not crash is just that after the first
>              iteration the results are different (wrong in the command
>         line).
>
>              The solution I am getting in eclipse, for each iteration,
>         is the
>              same that I would get if running the algorithm in octave for
>              example, so I am sure the solution is correct.
>
>              I have tried using java plus the command-line arguments of
>         eclipse
>              on the command line and that also works ok (local in ubuntu).
>
>              Has anybody experienced something similar? Any idea why
>         this could
>              happen? how can I fix this?
>
>              Regards,
>              Marcela.
>
>
>
>


-- 
Dr. Marcela Charfuelan, Senior Researcher
TU Berlin, School of Electrical Engineering and Computer Sciences
Database Systems and Information Management (DIMA)
EN7, Einsteinufer 17, D-10587 Berlin
Room: EN 725  Phone: +49 30-314-23556
URL: http://www.user.tu-berlin.de/charfuelan

Mime
View raw message