spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-18612][MLLIB] Delete broadcasted variable in LBFGS CostFun
Date Wed, 30 Nov 2016 10:04:52 GMT
Repository: spark
Updated Branches:
  refs/heads/master fe854f2e4 -> c5a64d760


[SPARK-18612][MLLIB] Delete broadcasted variable in LBFGS CostFun

## What changes were proposed in this pull request?

Fix a broadcasted variable leak occurring at each invocation of CostFun in L-BFGS.

## How was this patch tested?

UTests + check that fixed fatal memory consumption on Criteo's use cases.

This contribution is made on behalf of Criteo S.A.
(http://labs.criteo.com/) under the terms of the Apache v2 License.

Author: Anthony Truchet <a.truchet@criteo.com>

Closes #16040 from AnthonyTruchet/SPARK-18612-lbfgs-cost-fun.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c5a64d76
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c5a64d76
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c5a64d76

Branch: refs/heads/master
Commit: c5a64d760600ff430899e401751c41dc6b27cee6
Parents: fe854f2
Author: Anthony Truchet <a.truchet@criteo.com>
Authored: Wed Nov 30 10:04:47 2016 +0000
Committer: Sean Owen <sowen@cloudera.com>
Committed: Wed Nov 30 10:04:47 2016 +0000

----------------------------------------------------------------------
 .../main/scala/org/apache/spark/mllib/optimization/LBFGS.scala    | 3 +++
 1 file changed, 3 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/c5a64d76/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala b/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala
index 900eec1..e0e41f7 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/optimization/LBFGS.scala
@@ -252,6 +252,9 @@ object LBFGS extends Logging {
             (grad1, loss1 + loss2)
           })
 
+      // broadcasted model is not needed anymore
+      bcW.destroy()
+
       /**
        * regVal is sum of weight squares if it's L2 updater;
        * for other updater, the same logic is followed.


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message