systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Boehm (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SYSTEMML-2162) Deserialization of compressed blocks w/ shared dictionary fails in Spark
Date Wed, 28 Feb 2018 02:31:00 GMT
Matthias Boehm created SYSTEMML-2162:
----------------------------------------

             Summary: Deserialization of compressed blocks w/ shared dictionary fails in Spark
                 Key: SYSTEMML-2162
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2162
             Project: SystemML
          Issue Type: Bug
            Reporter: Matthias Boehm


For very large datasets that exceed aggregate cluster memory, we automatically apply compression.
However, in case blocks with shared DDC1 dictionaries are evicted and subsequently deserialized
this deserialization fails with mismatching number of bytes.

{code}
java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
{code}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message