spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanbo Liang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-15605) ML JavaDeveloperApiExample is broken
Date Wed, 01 Jun 2016 17:16:59 GMT

     [ https://issues.apache.org/jira/browse/SPARK-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yanbo Liang updated SPARK-15605:
--------------------------------
    Description: 
This bug is reported by
http://apache-spark-developers-list.1001551.n3.nabble.com/Creation-of-SparkML-Estimators-in-Java-broken-td17710.html
.



  was:
This bug is reported by
http://apache-spark-developers-list.1001551.n3.nabble.com/Creation-of-SparkML-Estimators-in-Java-broken-td17710.html
.

To fix the issue mentioned in the mailing list, we can implement maxIter as following:
{code}
  private IntParam maxIter_;

  public IntParam maxIter() {
    return maxIter_;
  }

  public int getMaxIter() {
    return (Integer) getOrDefault(maxIter_);
  }

  public MyJavaLogisticRegression setMaxIter(int value) {
    return (MyJavaLogisticRegression) set(maxIter_, value);
  }

  private void init() {
    maxIter_ = new IntParam(this, "maxIter", "max number of iterations");
    setDefault(maxIter(), 100);
  }
{code}

But this does not solve all problems, it will throw new exceptions:
{code}
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Param null__featuresCol
does not belong to myJavaLogReg_fbcf0d015036.
{code}
The shared params such as "featuresCol" should also have Java-friendly wrappers.
Look through the code base, I found we only implement JavaParams who is the wrappers of Scala
Params.
We still need Java-friendly wrappers for other traits who extends from Scala Params.

For example, in Scala we have: 
{code}
trait HasLabelCol extends Params
{code}
We should have the Java-friendly wrappers as follows:
{code}
class JavaHasLabelCol extends Params
{code}

Otherwise, "Param.parent" will be null for each param and it will throw exceptions when calling
"Param.hasParam()".

I think the Java compatibility for Params/Param should be further defined in the next release
cycle, and it's better to remove the "JavaDeveloperApiExample" at present. Since currently
we can not well support users to implement their own algorithm using Estimator/Transformer
by Java, it may mislead users.


> ML JavaDeveloperApiExample is broken
> ------------------------------------
>
>                 Key: SPARK-15605
>                 URL: https://issues.apache.org/jira/browse/SPARK-15605
>             Project: Spark
>          Issue Type: Bug
>          Components: Examples, ML
>    Affects Versions: 2.0.0
>            Reporter: Yanbo Liang
>
> This bug is reported by
> http://apache-spark-developers-list.1001551.n3.nabble.com/Creation-of-SparkML-Estimators-in-Java-broken-td17710.html
.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message