avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Kincaid <kincaid.d...@gmail.com>
Subject OutOfMemoryError while writing a large map to a file
Date Thu, 28 Jan 2016 15:51:02 GMT
I am getting an OutOfMemoryError while writing a very large map to a file.
It appears that it is trying to grow the size of the buffer at some point
and trying to make a copy of the data already in the buffer, but I could be
misinterpreting the stack trace. I have the JVM heap set to 24gb right now
and can't really go any higher.

I'm using the Clojure library "abracad" to do the writing. It's using a
DataFileWriter and ClojureDatumWriter which is a subclass of
GenericDatumWriter. I have a Clojure map that I am simply passing to the
append method of the DataFileWriter. Does anyone have a suggestion for
making the write not try to copy the data in memory as it's writing?

Here is the relevant piece of the stack trace:

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
        at java.util.Arrays.copyOf(Arrays.java:3236)
        at
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        at
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        at
org.apache.avro.io.BufferedBinaryEncoder$OutputStreamSink.innerWrite(BufferedBinaryEncoder.java:216)
        at
org.apache.avro.io.BufferedBinaryEncoder.flushBuffer(BufferedBinaryEncoder.java:93)
        at
org.apache.avro.io.BufferedBinaryEncoder.ensureBounds(BufferedBinaryEncoder.java:108)
        at
org.apache.avro.io.BufferedBinaryEncoder.writeLong(BufferedBinaryEncoder.java:129)
        at abracad.avro.ClojureDatumWriter.write(ClojureDatumWriter.java:47)
        at
org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:180)
        at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:69)
        at abracad.avro.ClojureDatumWriter.write(ClojureDatumWriter.java:51)
        at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
        at
org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)
        at
com.idexx.medical_notes.read_notes$map__GT_avro_file.invoke(read_notes.clj:46)
        at
com.idexx.medical_notes.read_notes$listener$fn__5159$fn__5161.invoke(read_notes.clj:244)
        at
com.idexx.medical_notes.read_notes.proxy$akka.actor.UntypedActor$ff19274a.onReceive(Unknown
Source)
        at
akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
        at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
        at
com.idexx.medical_notes.read_notes.proxy$akka.actor.UntypedActor$ff19274a.aroundReceive(Unknown
Source)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
        at akka.actor.ActorCell.invoke(ActorCell.scala:487)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
        at akka.dispatch.Mailbox.run(Mailbox.scala:220)
        at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
        at abracad.avro.ClojureDatumWriter.write(ClojureDatumWriter.java:47)
        at
org.apache.avro.generic.GenericDatumWriter.writeMap(GenericDatumWriter.java:180)
        at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:69)
        at abracad.avro.ClojureDatumWriter.write(ClojureDatumWriter.java:51)
        at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
        at
org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290)

Mime
View raw message