Mailing-List: contact user-help@predictionio.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@predictionio.apache.org
From: LiJunjie <holysoros@gmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Subject: Failed to persist models with java.lang.NegativeArraySizeException
 exception in kryo
Message-Id: <204D5E99-F4D6-4315-96EB-9385F9B0D5D9@gmail.com>
Date: Wed, 6 Dec 2017 16:39:59 +0800
To: user@predictionio.incubator.apache.org
archived-at: Wed, 06 Dec 2017 08:40:08 -0000

Hi all,

I have successfully trained, but failed to persist models. I notice =
PredictionIO will serialize models with kryo into Array[Byte], then what =
will happen if the trained models are very big. I also noticed the =
driver JVM heap grow rapidly at that time.

I use the official e-commerce recommender template, and the model to be =
serialized like this:

 class ECommModel(
   val rank: Int,
   val userFeatures: Map[Int, Array[Double]],
   val productModels: Map[Int, ProductModel],
   val userStringIntMap: BiMap[String, Int],
   val itemStringIntMap: BiMap[String, Int]
 ) extends Serializable

But there are several millions of records in userFeatures and =
productModels!

Here is the backtrace:

[INFO] [Engine$] EngineWorkflow.train completed
[INFO] [Engine] engineInstanceId=3D9f31aafc-2d66-4b76-9995-ad3e9fb84b6b
[INFO] [CoreWorkflow$] Inserting persistent model


[INFO] [AbstractConnector] Stopped =
Spark@5d1f640c{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
Exception in thread "main" com.esotericsoftware.kryo.KryoException: =
java.lang.NegativeArraySizeException
Serialization trace:
userFeatures (org.example.ecommercerecommendation.ECommModel)
        at =
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:1=
01)
        at =
com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerialize=
r.java:518)
        at =
com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
        at =
com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable=
.scala:29)
        at =
com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable=
.scala:27)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at =
scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at =
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
        at =
com.twitter.chill.TraversableSerializer.write(Traversable.scala:27)
        at =
com.twitter.chill.TraversableSerializer.write(Traversable.scala:21)
        at =
com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
        at =
com.twitter.chill.SerDeState.writeClassAndObject(SerDeState.java:64)
        at =
com.twitter.chill.KryoPool.toBytesWithClass(KryoPool.java:116)
        at =
com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:64)
        at =
com.twitter.chill.KryoInjectionInstance.apply(KryoInjection.scala:55)
        at =
org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala=
:81)
        at =
org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala=
:251)
        at =
org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)=

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:=
62)
        at =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm=
pl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at =
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$=
runMain(SparkSubmit.scala:755)
        at =
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at =
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at =
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NegativeArraySizeException
        at =
com.esotericsoftware.kryo.util.IdentityObjectIntMap.resize(IdentityObjectI=
ntMap.java:447)
        at =
com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjec=
tIntMap.java:245)
        at =
com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectInt=
Map.java:239)
        at =
com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntM=
ap.java:135)
        at =
com.esotericsoftware.kryo.util.IdentityObjectIntMap.putStash(IdentityObjec=
tIntMap.java:246)
        at =
com.esotericsoftware.kryo.util.IdentityObjectIntMap.push(IdentityObjectInt=
Map.java:239)
        at =
com.esotericsoftware.kryo.util.IdentityObjectIntMap.put(IdentityObjectIntM=
ap.java:135)
        at =
com.esotericsoftware.kryo.util.MapReferenceResolver.addWrittenObject(MapRe=
ferenceResolver.java:41)
        at =
com.esotericsoftware.kryo.Kryo.writeReferenceOrNull(Kryo.java:658)
        at =
com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:623)
        at =
com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:37)
        at =
com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
        at =
com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628)
        at =
com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable=
.scala:29)
        at =
com.twitter.chill.TraversableSerializer$$anonfun$write$1.apply(Traversable=
.scala:27)
        at =
scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
        at =
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at =
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at =
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at =
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at =
scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
        at =
com.twitter.chill.TraversableSerializer.write(Traversable.scala:27)
        at =
com.twitter.chill.TraversableSerializer.write(Traversable.scala:21)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:552)
        at =
com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:8=
0)
        ... 27 more