flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thang Nguyen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1733) Add PCA to machine learning library
Date Fri, 22 Jan 2016 21:25:40 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15113112#comment-15113112

Thang Nguyen commented on FLINK-1733:

Hi [~till.rohrmann]! 

I am a software engineer professionally, however I am new to Scala. I did learn some functional
programming in undergrad, so the trickiest thing for me to wrap my head around is Scala's
type system. 

For context: 
I have a naive PCA implementation and some trivial tests for it (using the method & test
data from this paper: http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf).

Currently, the method accepts an Int (# of princ. components), and a DataSet[DenseVector].

For this implementation, I create a covariance matrix (BreezeMatrix) and call breeze.linalg.eigSym
on that. 
Then I return the top N (user param) principal components as a DataSet[Vector].
I will be re-factoring/throwing out a lot of my code (except the tests), so I hesitate to
show anything I've written just yet.

Does the method signature make sense? 
What _exactly_ should I be returning?  The concept of PCA is new to me but it sounds like
I should be returning the top N vectors (based on their eigenvalues, ordered by significance).

Should the output also be DataSet[DenseVector]?
Pointers on how to implement sPCA? 

I have taken a cursory look at the rest of the ML library but I am still learning Scala. 
If you have any recommended resources on learning Scala (specifically the type system), I
would also appreciate that. 



> Add PCA to machine learning library
> -----------------------------------
>                 Key: FLINK-1733
>                 URL: https://issues.apache.org/jira/browse/FLINK-1733
>             Project: Flink
>          Issue Type: New Feature
>          Components: Machine Learning Library
>            Reporter: Till Rohrmann
>            Assignee: Thang Nguyen
>            Priority: Minor
>              Labels: ML
> Dimension reduction is a crucial prerequisite for many data analysis tasks. Therefore,
Flink's machine learning library should contain a principal components analysis (PCA) implementation.
Maria-Florina Balcan et al. [1] proposes a distributed PCA. A more recent publication [2]
describes another scalable PCA implementation.
> Resources:
> [1] [http://arxiv.org/pdf/1408.5823v5.pdf]
> [2] [http://ds.qcri.org/images/profile/tarek_elgamal/sigmod2015.pdf]

This message was sent by Atlassian JIRA

View raw message