spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-26326) Cannot save a NaiveBayesModel with 48685 features and 5453 labels
Date Fri, 01 Mar 2019 20:41:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-26326:
------------------------------
    Priority: Minor  (was: Major)

Yeah, this means you have a model with about 265M parameters, and when serialized as an array
of bytes, is (barely) exceeding 2GB. I think reimplementing this under the hood is possible
but it may call into question whether this is a realistic use case for naive bayes?

> Cannot save a NaiveBayesModel with 48685 features and 5453 labels
> -----------------------------------------------------------------
>
>                 Key: SPARK-26326
>                 URL: https://issues.apache.org/jira/browse/SPARK-26326
>             Project: Spark
>          Issue Type: Bug
>          Components: ML
>    Affects Versions: 2.4.0
>            Reporter: Markus Paaso
>            Priority: Minor
>
> When executing
> {code:java}
> model.write().overwrite().save("/tmp/mymodel"){code}
> The error occurs
> {code:java}
> java.lang.UnsupportedOperationException: Cannot convert this array to unsafe format as
it's too big.
> at org.apache.spark.sql.catalyst.expressions.UnsafeArrayData.fromPrimitiveArray(UnsafeArrayData.java:457)
> at org.apache.spark.sql.catalyst.expressions.UnsafeArrayData.fromPrimitiveArray(UnsafeArrayData.java:524)
> at org.apache.spark.ml.linalg.MatrixUDT.serialize(MatrixUDT.scala:66)
> at org.apache.spark.ml.linalg.MatrixUDT.serialize(MatrixUDT.scala:28)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$UDTConverter.toCatalystImpl(CatalystTypeConverters.scala:143)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:103)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:258)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:238)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:103)
> at org.apache.spark.sql.catalyst.CatalystTypeConverters$.$anonfun$createToCatalystConverter$2(CatalystTypeConverters.scala:396)
> at org.apache.spark.sql.catalyst.plans.logical.LocalRelation$.$anonfun$fromProduct$1(LocalRelation.scala:43)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
> at scala.collection.immutable.List.foreach(List.scala:388)
> at scala.collection.TraversableLike.map(TraversableLike.scala:233)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
> at scala.collection.immutable.List.map(List.scala:294)
> at org.apache.spark.sql.catalyst.plans.logical.LocalRelation$.fromProduct(LocalRelation.scala:43)
> at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:315)
> at org.apache.spark.ml.classification.NaiveBayesModel$NaiveBayesModelWriter.saveImpl(NaiveBayes.scala:393)
> at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:180)
> {code}
> Data file to reproduce the problem: [https://github.com/make/spark-26326-files/raw/master/data.libsvm]
> Code to reproduce the problem:
> {code:java}
> import org.apache.spark.ml.classification.NaiveBayes
> import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
> // Load the data stored in LIBSVM format as a DataFrame.
> val data = spark.read.format("libsvm").load("/tmp/data.libsvm")
> // Train a NaiveBayes model.
> val model = new NaiveBayes().fit(data)
> model.write().overwrite().save("/tmp/mymodel"){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message