spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-16470][ML][OPTIMIZER] Check linear regression training whether actually reach convergence and add warning if not
Date Tue, 12 Jul 2016 12:04:37 GMT
Repository: spark
Updated Branches:
  refs/heads/master 5b28e0258 -> 6cb75db9a


[SPARK-16470][ML][OPTIMIZER] Check linear regression training whether actually reach convergence
and add warning if not

## What changes were proposed in this pull request?

In `ml.regression.LinearRegression`, it use breeze `LBFGS` and `OWLQN` optimizer to do data
training, but do not check whether breeze's optimizer returned result actually reached convergence.

The `LBFGS` and `OWLQN` optimizer in breeze finish iteration may result the following situations:

1) reach max iteration number
2) function reach value convergence
3) objective function stop improving
4) gradient reach convergence
5) search failed(due to some internal numerical error)

I add warning printing code so that
if the iteration result is (1) or (3) or (5) in above, it will print a warning with respective
reason string.

## How was this patch tested?

Manual.

Author: WeichenXu <WeichenXu123@outlook.com>

Closes #14122 from WeichenXu123/add_lr_not_convergence_warn.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6cb75db9
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6cb75db9
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6cb75db9

Branch: refs/heads/master
Commit: 6cb75db9ab1a4f227069bec2763b89546b88b0ee
Parents: 5b28e02
Author: WeichenXu <WeichenXu123@outlook.com>
Authored: Tue Jul 12 13:04:34 2016 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Tue Jul 12 13:04:34 2016 +0100

----------------------------------------------------------------------
 .../scala/org/apache/spark/ml/regression/LinearRegression.scala | 5 +++++
 1 file changed, 5 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/6cb75db9/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
index 0477f71..6b82ae1 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala
@@ -327,6 +327,11 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0") override val
uid: String
         throw new SparkException(msg)
       }
 
+      if (!state.actuallyConverged) {
+        logWarning("LinearRegression training fininshed but the result " +
+          s"is not converged because: ${state.convergedReason.get.reason}")
+      }
+
       /*
          The coefficients are trained in the scaled space; we're converting them back to
          the original space.


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message