spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yanbo Liang (JIRA)" <>
Subject [jira] [Updated] (SPARK-15605) ML JavaDeveloperApiExample is broken
Date Wed, 01 Jun 2016 17:16:59 GMT


Yanbo Liang updated SPARK-15605:
This bug is reported by

This bug is reported by

To fix the issue mentioned in the mailing list, we can implement maxIter as following:
  private IntParam maxIter_;

  public IntParam maxIter() {
    return maxIter_;

  public int getMaxIter() {
    return (Integer) getOrDefault(maxIter_);

  public MyJavaLogisticRegression setMaxIter(int value) {
    return (MyJavaLogisticRegression) set(maxIter_, value);

  private void init() {
    maxIter_ = new IntParam(this, "maxIter", "max number of iterations");
    setDefault(maxIter(), 100);

But this does not solve all problems, it will throw new exceptions:
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Param null__featuresCol
does not belong to myJavaLogReg_fbcf0d015036.
The shared params such as "featuresCol" should also have Java-friendly wrappers.
Look through the code base, I found we only implement JavaParams who is the wrappers of Scala
We still need Java-friendly wrappers for other traits who extends from Scala Params.

For example, in Scala we have: 
trait HasLabelCol extends Params
We should have the Java-friendly wrappers as follows:
class JavaHasLabelCol extends Params

Otherwise, "Param.parent" will be null for each param and it will throw exceptions when calling

I think the Java compatibility for Params/Param should be further defined in the next release
cycle, and it's better to remove the "JavaDeveloperApiExample" at present. Since currently
we can not well support users to implement their own algorithm using Estimator/Transformer
by Java, it may mislead users.

> ML JavaDeveloperApiExample is broken
> ------------------------------------
>                 Key: SPARK-15605
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Examples, ML
>    Affects Versions: 2.0.0
>            Reporter: Yanbo Liang
> This bug is reported by

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message