spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hollin Wilkins (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-13944) Separate out local linear algebra as a standalone module without Spark dependency
Date Sat, 30 Apr 2016 02:09:12 GMT

    [ https://issues.apache.org/jira/browse/SPARK-13944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265076#comment-15265076
] 

Hollin Wilkins commented on SPARK-13944:
----------------------------------------

Is there any work being done on implementing core model logic in this module? For instance,
adding a LinearRegressionModel class to it that does simple scoring for linear regression,
or random forest models? This would be very useful as a standalone ML library with a nice
portable linear algebra library underpinning it.

> Separate out local linear algebra as a standalone module without Spark dependency
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-13944
>                 URL: https://issues.apache.org/jira/browse/SPARK-13944
>             Project: Spark
>          Issue Type: New Feature
>          Components: Build, ML
>    Affects Versions: 2.0.0
>            Reporter: Xiangrui Meng
>            Assignee: DB Tsai
>            Priority: Blocker
>
> Separate out linear algebra as a standalone module without Spark dependency to simplify
production deployment. We can call the new module spark-mllib-local, which might contain local
models in the future.
> The major issue is to remove dependencies on user-defined types.
> The package name will be changed from mllib to ml. For example, Vector will be changed
from `org.apache.spark.mllib.linalg.Vector` to `org.apache.spark.ml.linalg.Vector`. The return
vector type in the new ML pipeline will be the one in ML package; however, the existing mllib
code will not be touched. As a result, this will potentially break the API. Also, when the
vector is loaded from mllib vector by Spark SQL, the vector will automatically converted into
the one in ml package.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message