flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <fhue...@gmail.com>
Subject Re: Information on Flink
Date Thu, 13 Nov 2014 20:44:36 GMT
Hi Andrea,


Please correct me if I got something wrong.

You have a population of several classifiers that you want to evolve and improve.

In each iteration, you test each classifier with all your test data and evaluate the fitness
of each classifier. Then, you create a new population of classifiers by removing bad classifiers
and mutating (and crossing) the better ones.


I think you can do that with Flink as follows:

you use a Map over the test data and a broadcast set as the classifiers to check the classification
of a single attribute. With a following reduce, you aggregate the fitness of each classifier
and use a reduce all to build a new population of classifiers.

In the next iteration, the new population is broadcasted to the Map over the test data.


This is quite similar to what our KMeans example does. You should have a look at it.


Best, Fabian





From: Kostas Tzoumas
Sent: ‎Thursday‎, ‎13‎. ‎November‎, ‎2014 ‎17‎:‎11
To: user@flink.incubator.apache.org
Cc: Andrea Ferranti






I am forwarding this here in case someone with better knowledge of genetic algorithms picks
it up.




Kostas


---------- Forwarded message ----------
From: Andrea Ferranti <andreaferranti@hotmail.it>
Date: Thu, Nov 13, 2014 at 4:46 PM
Subject: Re: Information on Flink
To: Kostas Tzoumas <ktzoumas@apache.org>




Thanks very much for your reply.







First, can I forward this to the Flink user mailing list? Perhaps someone over there has a
better answer.






Yes, of course.







Can you describe very briefly how fitness evaluation is computed in your algorithm?






My fitness evaluation is basically an evaluation of accuracy in a classification problem,
so i must read every line of file(in which is present some attribute and a class) and verify
if my classification work well.

So in each iteration of a genetic algorithm i change some chromosome and than evaluate the
solution.




At the moment the entire program is written in C++ but I would take it in java using jMetal




Best regards, 

Andrea




Il giorno 13/nov/2014, alle ore 16:25, Kostas Tzoumas <ktzoumas@apache.org> ha scritto:





Hey,



First, can I forward this to the Flink user mailing list? Perhaps someone over there has a
better answer.




Can you describe very briefly how fitness evaluation is computed in your algorithm?




Kostas



On Thu, Nov 13, 2014 at 4:08 PM, Andrea Ferranti <andreaferranti@hotmail.it> wrote:


Dear Kostas Tzoumas,
I'm Andrea Ferranti, a student of Computer Engineering at the University of Pisa.


In my thesis I would like to exploit Flink to parallelize a Evolutionary algorithm, in particular
the fitness evaluation. My problem and algorithm are written in Java (jMetal).







Do you think that flink can be a good tool for the parallelization of fitness? In my problem
the fitness is evaluate on very big datasets.




Best regards, 

Andrea
Mime
View raw message