griffin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [griffin] wankunde opened a new pull request #533: [GRIFFIN-288]optimize hdfs sink
Date Fri, 06 Sep 2019 11:42:06 GMT
wankunde opened a new pull request #533: [GRIFFIN-288]optimize hdfs sink
URL: https://github.com/apache/griffin/pull/533
 
 
   When we sink records to hdfs , it may be OOM if the result is huge.
   
   ```
   19/09/06 18:52:39 INFO LineBufferedStream: 19/09/06 18:52:39 ERROR sink.HdfsSink: Java
heap space
   19/09/06 18:52:39 INFO LineBufferedStream: java.lang.OutOfMemoryError: Java heap space
   19/09/06 18:52:39 INFO LineBufferedStream:      at java.util.Arrays.copyOf(Arrays.java:3332)
   19/09/06 18:52:39 INFO LineBufferedStream:      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
   19/09/06 18:52:39 INFO LineBufferedStream:      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
   19/09/06 18:52:39 INFO LineBufferedStream:      at java.lang.StringBuilder.append(StringBuilder.java:136)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:200)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.TraversableOnce$$anonfun$addString$1.apply(TraversableOnce.scala:364)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:357)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.AbstractTraversable.addString(Traversable.scala:104)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:323)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.AbstractTraversable.mkString(Traversable.scala:104)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:325)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.AbstractTraversable.mkString(Traversable.scala:104)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.sink.HdfsSink.org$apache$griffin$measure$sink$HdfsSink$$sinkRecords2Hdfs(HdfsSink.scala:191)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.sink.HdfsSink.sinkRecords(HdfsSink.scala:133)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.sink.MultiSinks$$anonfun$sinkRecords$1.apply(MultiSinks.scala:63)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.sink.MultiSinks$$anonfun$sinkRecords$1.apply(MultiSinks.scala:61)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.collection.immutable.List.foreach(List.scala:392)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.sink.MultiSinks.sinkRecords(MultiSinks.scala:61)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.step.write.RecordWriteStep.execute(RecordWriteStep.scala:49)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.step.transform.SparkSqlTransformStep.doExecute(SparkSqlTransformStep.scala:40)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.step.transform.TransformStep$class.execute(TransformStep.scala:72)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.step.transform.SparkSqlTransformStep.execute(SparkSqlTransformStep.scala:27)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.step.transform.TransformStep$$anonfun$2$$anonfun$apply$1.apply$mcV$sp(TransformStep.scala:51)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.step.transform.TransformStep$$anonfun$2$$anonfun$apply$1.apply(TransformStep.scala:50)
   19/09/06 18:52:39 INFO LineBufferedStream:      at org.apache.griffin.measure.step.transform.TransformStep$$anonfun$2$$anonfun$apply$1.apply(TransformStep.scala:50)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
   19/09/06 18:52:39 INFO LineBufferedStream:      at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
   19/09/06 18:52:39 INFO LineBufferedStream:      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   19/09/06 18:52:39 INFO LineBufferedStream:      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   19/09/06 18:52:39 INFO LineBufferedStream:      at java.lang.Thread.run(Thread.java:748)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message