spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <>
Subject [jira] [Commented] (SPARK-21535) Reduce memory requirement for CrossValidator and TrainValidationSplit
Date Wed, 02 Aug 2017 19:19:03 GMT


Hyukjin Kwon commented on SPARK-21535:

User 'hhbyyh' has created a pull request for this issue:

> Reduce memory requirement for CrossValidator and TrainValidationSplit 
> ----------------------------------------------------------------------
>                 Key: SPARK-21535
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML
>    Affects Versions: 2.2.0
>            Reporter: yuhao yang
> CrossValidator and TrainValidationSplit both use 
> {code}models =, epm) {code} to fit the models, where epm is Array[ParamMap].
> Even though the training process is sequential, current implementation consumes extra
driver memory for holding the trained models, which is not necessary and often leads to memory
exception for both CrossValidator and TrainValidationSplit. My proposal is to optimize the
training implementation, thus that used model can be collected by GC, and avoid the unnecessary
OOM exceptions.
> E.g. when grid search space is 12, old implementation needs to hold all 12 trained models
in the driver memory at the same time, while the new implementation only needs to hold 1 trained
model at a time, and previous model can be cleared by GC.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message