spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vincent (JIRA)" <>
Subject [jira] [Commented] (SPARK-21688) performance improvement in mllib SVM with native BLAS
Date Thu, 10 Aug 2017 13:46:00 GMT


Vincent commented on SPARK-21688:

Okay. Yes, true. It can still run without issue but we are just offering another choice for
those who wanna have 50% speedup or more by using native BLAS in their case, they can also
stick to F2J with a simple setting in spark configuration.

the problem for default thread settings has been discussed in
I believe it's non-trivial but seems it's a common issue for all native blas implementations,
there's not a good solution to this issue for now.

> performance improvement in mllib SVM with native BLAS 
> ------------------------------------------------------
>                 Key: SPARK-21688
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>    Affects Versions: 2.2.0
>         Environment: 4 nodes: 1 master node, 3 worker nodes
> model name      : Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
> Memory : 180G
> num of core per node: 10
>            Reporter: Vincent
>         Attachments: ddot unitest.png, mllib svm training.png, native-trywait.png, svm1.png,
svm2.png, svm-mkl-1.png, svm-mkl-2.png
> in current mllib SVM implementation, we found that the CPU is not fully utilized, one
reason is that f2j blas is set to be used in the HingeGradient computation. As we found out
earlier ( that with proper settings, native
blas is generally better than f2j on the uni-test level, here we make the blas operations
in SVM go with MKL blas and get an end to end performance report showing that in most cases
native blas outperformance f2j blas up to 50%.
> So, we suggest removing those f2j-fixed calling and going for native blas if available.
If this proposal is acceptable, we will move on to benchmark other algorithms impacted. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message