systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SYSTEMML-1762) Improve the matrix reshape function for the Spark mode
Date Thu, 13 Jul 2017 06:23:00 GMT

    [ https://issues.apache.org/jira/browse/SYSTEMML-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085249#comment-16085249
] 

Matthias Boehm commented on SYSTEMML-1762:
------------------------------------------

thanks [~Tenma] for catching this issue. As it turned out, this issue occurs in the special
case if, for a given input block, we create at least three output blocks and the first and
last output block have the same row index. For example, if we have an output matrix of 13
column blocks and we computed (1,12) and (1,1) as the first and last output block index, we
missed the middle index (1,13).

> Improve the matrix reshape function for the Spark mode
> ------------------------------------------------------
>
>                 Key: SYSTEMML-1762
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1762
>             Project: SystemML
>          Issue Type: Bug
>          Components: Algorithms, ParFor, Runtime
>            Reporter: Fei Hu
>            Assignee: Fei Hu
>         Attachments: MNIST_Distrib_Sgd.scala
>
>
> When running the [distributed MNIST LeNet example | https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml],
it works well in the hybrid mode. But in the Spark mode, there are some errors about
> {{java.lang.NullPointerException}}  and {{java.lang.ArrayIndexOutOfBoundsException: 1000}}
when reshaping the matrix. The involved functions are {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#reshapeSparse}}
and {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#reshapeDense}}. The reason is that
the output matrix index computed by {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#computeResultBlockIndex}}
does not match the keys in the {{HashMap<MatrixIndexes,MatrixBlock> rix}}. 
> To reproduce the error, the attached scala file {{MNIST_Distrib_Sgd.scala}} could be
used to run the distributed MNIST example.  
> In addition, if adding some codes to ignore the null output matrix block from {{MatrixBlock
out = rix.get(ixtmp)}},  the distributed MNIST example could run in the Spark mode, but the
result may not be right. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message