spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-20687) mllib.Matrices.fromBreeze may crash when converting breeze CSCMatrix
Date Thu, 11 May 2017 17:21:04 GMT

    [ https://issues.apache.org/jira/browse/SPARK-20687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006804#comment-16006804
] 

Apache Spark commented on SPARK-20687:
--------------------------------------

User 'ghoto' has created a pull request for this issue:
https://github.com/apache/spark/pull/17940

> mllib.Matrices.fromBreeze may crash when converting breeze CSCMatrix
> --------------------------------------------------------------------
>
>                 Key: SPARK-20687
>                 URL: https://issues.apache.org/jira/browse/SPARK-20687
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 2.1.1
>            Reporter: Ignacio Bermudez Corrales
>            Priority: Minor
>
> Conversion of Breeze sparse matrices to Matrix is broken when matrices are product of
certain operations. This problem I think is caused by the update method in Breeze CSCMatrix
when they add provisional zeros to the data for efficiency.
> This bug is serious and may affect at least BlockMatrix addition and substraction
> http://stackoverflow.com/questions/33528555/error-thrown-when-using-blockmatrix-add/43883458#43883458
> The following code, reproduces the bug (Check test("breeze conversion bug"))
> https://github.com/ghoto/spark/blob/test-bug/CSCMatrixBreeze/mllib/src/test/scala/org/apache/spark/mllib/linalg/MatricesSuite.scala
> {code:title=MatricesSuite.scala|borderStyle=solid}
>   test("breeze conversion bug") {
>     // (2, 0, 0)
>     // (2, 0, 0)
>     val mat1Brz = Matrices.sparse(2, 3, Array(0, 2, 2, 2), Array(0, 1), Array(2, 2)).asBreeze
>     // (2, 1E-15, 1E-15)
>     // (2, 1E-15, 1E-15
>     val mat2Brz = Matrices.sparse(2, 3, Array(0, 2, 4, 6), Array(0, 0, 0, 1, 1, 1), Array(2,
1E-15, 1E-15, 2, 1E-15, 1E-15)).asBreeze
>     // The following shouldn't break
>     val t01 = mat1Brz - mat1Brz
>     val t02 = mat2Brz - mat2Brz
>     val t02Brz = Matrices.fromBreeze(t02)
>     val t01Brz = Matrices.fromBreeze(t01)
>     val t1Brz = mat1Brz - mat2Brz
>     val t2Brz = mat2Brz - mat1Brz
>     // The following ones should break
>     val t1 = Matrices.fromBreeze(t1Brz)
>     val t2 = Matrices.fromBreeze(t2Brz)
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message