spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From debasish83 <...@git.apache.org>
Subject [GitHub] spark pull request: [ML] SPARK-2426: Breeze NNLS integrated to ml ...
Date Wed, 18 Mar 2015 16:27:19 GMT
Github user debasish83 commented on the pull request:

    https://github.com/apache/spark/pull/5005#issuecomment-83045631
  
    I updated with the @mengxr checkin that cleaned jblas dependencies from NNLS....the results
updated above is with rank = 10 I think...A more realistic rank is 50...
    
    breeze nnls (including iterator overhead):
    
    15/03/18 09:21:45 INFO ALS: solveTime 177.048 ms
    15/03/18 09:21:45 INFO ALS: solveTime 189.642 ms
    15/03/18 09:21:45 INFO ALS: solveTime 171.894 ms
    15/03/18 09:21:45 INFO ALS: solveTime 203.503 ms
    15/03/18 09:21:45 INFO ALS: solveTime 185.352 ms
    15/03/18 09:21:45 INFO ALS: solveTime 190.051 ms
    15/03/18 09:21:45 INFO ALS: solveTime 181.634 ms
    15/03/18 09:21:45 INFO ALS: solveTime 148.325 ms
    15/03/18 09:21:46 INFO ALS: solveTime 59.523 ms
    15/03/18 09:21:46 INFO ALS: solveTime 78.501 ms
    15/03/18 09:21:46 INFO ALS: solveTime 66.649 ms
    15/03/18 09:21:46 INFO ALS: solveTime 66.078 ms
    15/03/18 09:21:46 INFO ALS: solveTime 75.827 ms
    15/03/18 09:21:46 INFO ALS: solveTime 50.72 ms
    15/03/18 09:21:46 INFO ALS: solveTime 65.047 ms
    15/03/18 09:21:46 INFO ALS: solveTime 59.184 ms
    15/03/18 09:21:46 INFO ALS: solveTime 26.837 ms
    15/03/18 09:21:46 INFO ALS: solveTime 26.061 ms
    15/03/18 09:21:46 INFO ALS: solveTime 36.14 ms
    15/03/18 09:21:46 INFO ALS: solveTime 33.341 ms
    15/03/18 09:21:46 INFO ALS: solveTime 29.399 ms
    15/03/18 09:21:46 INFO ALS: solveTime 28.61 ms
    15/03/18 09:21:46 INFO ALS: solveTime 30.074 ms
    15/03/18 09:21:46 INFO ALS: solveTime 32.351 ms
    15/03/18 09:21:46 INFO ALS: solveTime 56.625 ms
    15/03/18 09:21:47 INFO ALS: solveTime 54.29 ms
    15/03/18 09:21:47 INFO ALS: solveTime 49.305 ms
    15/03/18 09:21:47 INFO ALS: solveTime 63.742 ms
    15/03/18 09:21:47 INFO ALS: solveTime 67.626 ms
    15/03/18 09:21:47 INFO ALS: solveTime 55.348 ms
    15/03/18 09:21:47 INFO ALS: solveTime 54.473 ms
    15/03/18 09:21:47 INFO ALS: solveTime 66.704 ms
    15/03/18 09:21:47 INFO ALS: solveTime 29.57 ms
    15/03/18 09:21:47 INFO ALS: solveTime 33.871 ms
    15/03/18 09:21:47 INFO ALS: solveTime 33.329 ms
    15/03/18 09:21:47 INFO ALS: solveTime 40.322 ms
    15/03/18 09:21:47 INFO ALS: solveTime 35.376 ms
    15/03/18 09:21:47 INFO ALS: solveTime 35.826 ms
    15/03/18 09:21:47 INFO ALS: solveTime 38.973 ms
    15/03/18 09:21:47 INFO ALS: solveTime 34.908 ms
    
    breeze nnls inner solve (subtracting iterator time):
    
    15/03/18 09:21:45 INFO ALS: innerTime 103.235 ms
    15/03/18 09:21:45 INFO ALS: innerTime 115.842 ms
    15/03/18 09:21:45 INFO ALS: innerTime 104.233 ms
    15/03/18 09:21:45 INFO ALS: innerTime 112.84 ms
    15/03/18 09:21:45 INFO ALS: innerTime 97.474 ms
    15/03/18 09:21:45 INFO ALS: innerTime 91.188 ms
    15/03/18 09:21:45 INFO ALS: innerTime 115.043 ms
    15/03/18 09:21:45 INFO ALS: innerTime 96.443 ms
    15/03/18 09:21:46 INFO ALS: innerTime 31.696 ms
    15/03/18 09:21:46 INFO ALS: innerTime 40.446 ms
    15/03/18 09:21:46 INFO ALS: innerTime 42.288 ms
    15/03/18 09:21:46 INFO ALS: innerTime 34.147 ms
    15/03/18 09:21:46 INFO ALS: innerTime 33.404 ms
    15/03/18 09:21:46 INFO ALS: innerTime 28.02 ms
    15/03/18 09:21:46 INFO ALS: innerTime 33.372 ms
    15/03/18 09:21:46 INFO ALS: innerTime 38.531 ms
    15/03/18 09:21:46 INFO ALS: innerTime 20.651 ms
    15/03/18 09:21:46 INFO ALS: innerTime 19.721 ms
    15/03/18 09:21:46 INFO ALS: innerTime 27.865 ms
    15/03/18 09:21:46 INFO ALS: innerTime 21.655 ms
    15/03/18 09:21:46 INFO ALS: innerTime 22.413 ms
    15/03/18 09:21:46 INFO ALS: innerTime 19.793 ms
    15/03/18 09:21:46 INFO ALS: innerTime 18.858 ms
    15/03/18 09:21:46 INFO ALS: innerTime 24.089 ms
    15/03/18 09:21:46 INFO ALS: innerTime 43.602 ms
    15/03/18 09:21:47 INFO ALS: innerTime 44.16 ms
    15/03/18 09:21:47 INFO ALS: innerTime 32.279 ms
    15/03/18 09:21:47 INFO ALS: innerTime 53.446 ms
    15/03/18 09:21:47 INFO ALS: innerTime 48.279 ms
    15/03/18 09:21:47 INFO ALS: innerTime 44.157 ms
    15/03/18 09:21:47 INFO ALS: innerTime 37.386 ms
    15/03/18 09:21:47 INFO ALS: innerTime 56.11 ms
    15/03/18 09:21:47 INFO ALS: innerTime 21.043 ms
    15/03/18 09:21:47 INFO ALS: innerTime 25.038 ms
    15/03/18 09:21:47 INFO ALS: innerTime 24.97 ms
    15/03/18 09:21:47 INFO ALS: innerTime 27.335 ms
    15/03/18 09:21:47 INFO ALS: innerTime 26.706 ms
    15/03/18 09:21:47 INFO ALS: innerTime 27.418 ms
    15/03/18 09:21:47 INFO ALS: innerTime 30.885 ms
    15/03/18 09:21:47 INFO ALS: innerTime 19.971 ms
    
    mllib nnls:
    
    15/03/18 09:23:14 INFO ALS: solveTime 89.822 ms
    15/03/18 09:23:14 INFO ALS: solveTime 107.005 ms
    15/03/18 09:23:14 INFO ALS: solveTime 80.083 ms
    15/03/18 09:23:14 INFO ALS: solveTime 106.405 ms
    15/03/18 09:23:14 INFO ALS: solveTime 94.193 ms
    15/03/18 09:23:14 INFO ALS: solveTime 102.188 ms
    15/03/18 09:23:14 INFO ALS: solveTime 100.847 ms
    15/03/18 09:23:14 INFO ALS: solveTime 75.717 ms
    15/03/18 09:23:14 INFO ALS: solveTime 45.384 ms
    15/03/18 09:23:14 INFO ALS: solveTime 45.79 ms
    15/03/18 09:23:14 INFO ALS: solveTime 67.02 ms
    15/03/18 09:23:14 INFO ALS: solveTime 51.034 ms
    15/03/18 09:23:14 INFO ALS: solveTime 45.844 ms
    15/03/18 09:23:15 INFO ALS: solveTime 46.944 ms
    15/03/18 09:23:15 INFO ALS: solveTime 45.024 ms
    15/03/18 09:23:15 INFO ALS: solveTime 55.672 ms
    15/03/18 09:23:15 INFO ALS: solveTime 26.168 ms
    15/03/18 09:23:15 INFO ALS: solveTime 27.104 ms
    15/03/18 09:23:15 INFO ALS: solveTime 33.295 ms
    15/03/18 09:23:15 INFO ALS: solveTime 28.376 ms
    15/03/18 09:23:15 INFO ALS: solveTime 27.73 ms
    15/03/18 09:23:15 INFO ALS: solveTime 27.867 ms
    15/03/18 09:23:15 INFO ALS: solveTime 27.287 ms
    15/03/18 09:23:15 INFO ALS: solveTime 24.709 ms
    15/03/18 09:23:15 INFO ALS: solveTime 45.617 ms
    15/03/18 09:23:15 INFO ALS: solveTime 47.411 ms
    15/03/18 09:23:15 INFO ALS: solveTime 60.453 ms
    15/03/18 09:23:15 INFO ALS: solveTime 50.855 ms
    15/03/18 09:23:15 INFO ALS: solveTime 44.043 ms
    15/03/18 09:23:15 INFO ALS: solveTime 51.125 ms
    15/03/18 09:23:15 INFO ALS: solveTime 58.812 ms
    15/03/18 09:23:15 INFO ALS: solveTime 49.631 ms
    15/03/18 09:23:16 INFO ALS: solveTime 35.489 ms
    15/03/18 09:23:16 INFO ALS: solveTime 31.978 ms
    15/03/18 09:23:16 INFO ALS: solveTime 29.305 ms
    15/03/18 09:23:16 INFO ALS: solveTime 28.568 ms
    15/03/18 09:23:16 INFO ALS: solveTime 40.257 ms
    15/03/18 09:23:16 INFO ALS: solveTime 24.955 ms
    15/03/18 09:23:16 INFO ALS: solveTime 28.446 ms
    15/03/18 09:23:16 INFO ALS: solveTime 35.841 ms
    
    So at realistic ranks still there is an overhead from iterator but in overall flow I don't
know how much difference does this overhead causes...
    
    @mengxr if you have pointers on how to optimize Breeze NNLS code further please let me
know...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message