Hi Trevor,
great to hear that you have a working prototype :) And it is also good
that you shared your insights you gained when implementing it. Flink’s ML
library is far from perfect and, thus, all kinds of feedback is highly
valuable. In general it is always good to contribute code back if you think
they make a valuable addition. I try to give some comments to your points
as inline comments.
On Mon, Mar 28, 2016 at 8:01 PM, Trevor Grant <trevor.d.grant@gmail.com>
wrote:
Hey,
>
> I have a working prototype of an multi layer perceptron implementation
> working in Flink.
>
> I made every possible effort to utilize existing code when possible.
>
> In the process of doing this there were some hacks I want/need, and think
> this should be broken up into multiple PRs and possible abstract out the
> whole thing because the MLP implementation I came up with is itself
> designed to be extendable to Long Short Term Memory Networks.
> Top level here are some of the sub PRs
>
>  Expand SGD to allow for predicting vectors instead of just Doubles. This
> allows the same NN code (and other algos) to be used for classification,
> transformations, and regressions.
>
I agree that we could extend the LabeledVector to support a Vector[Double]
as label instead of a single Double. Initially we implemented it with a
single label value for the sake of simplicity. But I remember that we also
had a discussion about it. But somehow we didn’t derive any action points
from that. If you have code for that, then feel free to open a PR.
>  Allow for 'warm starts' > this requires adding a parameter to
> IterativeSolver that basically starts on iteration N. This is somewhat
> akin to the idea of partial fits in sklearn OR making the iterative solver
> have some sort of internal counter and then when you call 'fit' it just
> runs another N iterations (which is set by SetIterations) instead of
> assuming it is back to zero. This might seem trivial but has significant
> impact on step size calculations.
>
That is a good point and should not be too hard to add, I would assume.
>  A library of model grading metrics. Having 'calculate RSquare' as a built
> in method for every regressor doesn't seem like an efficient way to do this
> long term.
>
Agreed. The squaredResidualSum method of MLR is just a convenience method
to get at least one metric for the accuracy back. There is a PR open by
Theo which adds an evaluation framework to flinkML [1]. If I’m not mistaken
it should add some more generalized mean to calculate grading metrics.
> BLAS for matrix ops (this was talked about earlier)
>
Here the recommended way is to convert your matrix to a BreezeMatrix and
then use the BLAS operation from there.
>  A neural net has Arrays of matrices of weights (instead of just a
> vector). Currently I flatten the array of matrices out into a weight
> vector and reassemble it into an array of matrices, though this is probably
> not super effecient.
>
I would assume that you should simply operate on the flattened vector
without converting from one representation to the other.
>  The linear regression implementation currently presumes it will be using
> SGD but I think that should be 'settable' as a parameter, because if not
> why do we have all of those other nice SGD methods just hanging out?
> Similarly the loss function / partial loss is hard coded. I reccomend
> making the current setup the 'defaults' of a 'setOptimizer' method. I.e.
> if you want to just run a MLR you can do it based on the examples, but if
> you want to use a fancy optimizer you can create it from existing methods,
> or make your own, then call something like `mlr.setOptimizer( myOptimizer
> )`
>
> Agreed. That was actually also our plan but we haven’t come to that so
far. If you have the code available, then please open a PR for it.
 and more
>
> At any rate if some people could weigh in / direct me how to proceed that
> would be swell.
>
> Thanks!
> tg
>
>
>
>
> Trevor Grant
> Data Scientist
> https://github.com/rawkintrevo
> http://stackexchange.com/users/3002022/rawkintrevo
> http://trevorgrant.org
>
> *"Fortunate is he, who is able to know the causes of things." Virgil*
>
[1] https://github.com/apache/flink/pull/871
Cheers,
Till
