systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fei Hu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SYSTEMML-1774) Improve Parfor parallelism for deep learning
Date Mon, 17 Jul 2017 21:59:00 GMT

    [ https://issues.apache.org/jira/browse/SYSTEMML-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090327#comment-16090327
] 

Fei Hu edited comment on SYSTEMML-1774 at 7/17/17 9:58 PM:
-----------------------------------------------------------

The errors that running using the mode {{HYBRID_SPARK}} are different from that on the local
machine:

{code:java}
org.apache.sysml.api.mlcontext.MLContextException: Exception when executing script
  at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:320)
  at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:293)
  at org.apache.sysml.api.mlcontext.Script.execute(Script.java:717)
  at org.apache.sysml.scripts.nn.examples.Mnist_lenet_distrib_sgd.train(Mnist_lenet_distrib_sgd.java)
  ... 50 elided
Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception occurred while executing
runtime program
  at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:397)
  at org.apache.sysml.api.mlcontext.ScriptExecutor.execute(ScriptExecutor.java:311)
  at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:316)
  ... 53 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException:
ERROR: Runtime error in program block generated from statement block between lines 11 and
11 -- Error evaluating instruction: CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
  at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
  at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:95)
  at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:54)
  at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:395)
  ... 55 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block
generated from statement block between lines 11 and 11 -- Error evaluating instruction: CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
  at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
  ... 58 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: error executing function ./scripts/nn/examples/mnist_lenet_distrib_sgd.dml::train
  at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:187)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
  ... 61 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in function
program block generated from function statement block between lines 36 and 271 -- Error evaluating
function program block
  at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:122)
  at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:180)
  ... 62 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in for program
block generated from for statement block between lines 115 and 270 -- Error evaluating for
program block
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
  at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115)
  ... 63 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in for program
block generated from for statement block between lines 117 and 265 -- Error evaluating for
program block
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
  ... 64 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in
parallel.
  at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:660)
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
  ... 65 more
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage
2.0 failed 4 times, most recent failure: Lost task 3.3 in stage 2.0 (TID 20, rr-dense7.softlayer.com,
executor 6): org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program
block generated from statement block between lines 0 and 0 -- Error evaluating instruction:
CP°relu_backward°outa3·MATRIX·DOUBLE°_mVar3925·MATRIX·DOUBLE°_mVar3928·MATRIX·DOUBLE°-1
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
	at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeSetTask(ParWorker.java:167)
	at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeTask(ParWorker.java:136)
	at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:75)
	at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:40)
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at scala.collection.AbstractIterator.to(Iterator.scala:1336)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Error while executing multi-threaded
tasks
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:536)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.reluBackward(LibMatrixDNN.java:391)
	at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.processReluBackwardInstruction(ConvolutionCPInstruction.java:229)
	at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.proc
essInstruction(ConvolutionCPInstruction.java:316)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
	... 31 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:532)
	... 35 more
Caused by: java.lang.NullPointerException
	at org.apache.sysml.runtime.util.ConvolutionUtils.scalarOperations(ConvolutionUtils.java:150)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:231)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:208)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
  at scala.Option.foreach(Option.scala:257)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1925)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1938)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  at org.apache.spark.rdd.RDD.collect(RDD.scala:935)
  at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
  at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
  at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSpark.runJob(RemoteParForSpark.java:77)
  at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:1055)
  at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:647)
  ... 66 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block
generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°relu_backward°outa3·MATRIX·DOUBLE°_mVar3925·MATRIX·DOUBLE°_mVar3928·MATRIX·DOUBLE°-1
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
  at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeSetTask(ParWorker.java:167)
  at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeTask(ParWorker.java:136)
  at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:75)
  at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:40)
  at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
  at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
  at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
  at scala.collection.Iterator$class.foreach(Iterator.scala:893)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
  at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
  at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
  at scala.collection.AbstractIterator.to(Iterator.scala:1336)
  at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
  at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
  at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
  at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  at org.apache.spark.scheduler.Task.run(Task.scala:99)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
  ... 3 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Error while executing multi-threaded
tasks
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:536)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.reluBackward(LibMatrixDNN.java:391)
  at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.processReluBackwardInstruction(ConvolutionCPInstruction.java:229)
  at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.processInstruction(ConvolutionCPInstruction.java:316)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
  ... 31 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:532)
  ... 35 more
Caused by: java.lang.NullPointerException
  at org.apache.sysml.runtime.util.ConvolutionUtils.scalarOperations(ConvolutionUtils.java:150)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:231)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:208)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  ... 3 more
{code}



was (Author: tenma):
The errors that running on the Spark Cluster are different from that on the local machine:

{code:java}
org.apache.sysml.api.mlcontext.MLContextException: Exception when executing script
  at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:320)
  at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:293)
  at org.apache.sysml.api.mlcontext.Script.execute(Script.java:717)
  at org.apache.sysml.scripts.nn.examples.Mnist_lenet_distrib_sgd.train(Mnist_lenet_distrib_sgd.java)
  ... 50 elided
Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception occurred while executing
runtime program
  at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:397)
  at org.apache.sysml.api.mlcontext.ScriptExecutor.execute(ScriptExecutor.java:311)
  at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:316)
  ... 53 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException:
ERROR: Runtime error in program block generated from statement block between lines 11 and
11 -- Error evaluating instruction: CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
  at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
  at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:95)
  at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:54)
  at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:395)
  ... 55 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block
generated from statement block between lines 11 and 11 -- Error evaluating instruction: CP°extfunct°./scripts/nn/examples/mnist_lenet_distrib_sgd.dml°train°10°8°X·MATRIX·DOUBLE°Y·MATRIX·DOUBLE°X_val·MATRIX·DOUBLE°Y_val·MATRIX·DOUBLE°C·SCALAR·INT·false°Hin·SCALAR·INT·false°Win·SCALAR·INT·false°batch_size·SCALAR·INT·false°parallel_batches·SCALAR·INT·false°epochs·SCALAR·INT·false°W1°b1°W2°b2°W3°b3°W4°b4
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
  at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
  ... 58 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: error executing function ./scripts/nn/examples/mnist_lenet_distrib_sgd.dml::train
  at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:187)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
  ... 61 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in function
program block generated from function statement block between lines 36 and 271 -- Error evaluating
function program block
  at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:122)
  at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:180)
  ... 62 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in for program
block generated from for statement block between lines 115 and 270 -- Error evaluating for
program block
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
  at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115)
  ... 63 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in for program
block generated from for statement block between lines 117 and 265 -- Error evaluating for
program block
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:157)
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
  ... 64 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: PARFOR: Failed to execute loop in
parallel.
  at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:660)
  at org.apache.sysml.runtime.controlprogram.ForProgramBlock.execute(ForProgramBlock.java:145)
  ... 65 more
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage
2.0 failed 4 times, most recent failure: Lost task 3.3 in stage 2.0 (TID 20, rr-dense7.softlayer.com,
executor 6): org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program
block generated from statement block between lines 0 and 0 -- Error evaluating instruction:
CP°relu_backward°outa3·MATRIX·DOUBLE°_mVar3925·MATRIX·DOUBLE°_mVar3928·MATRIX·DOUBLE°-1
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
	at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeSetTask(ParWorker.java:167)
	at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeTask(ParWorker.java:136)
	at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:75)
	at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:40)
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
	at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
	at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at scala.collection.AbstractIterator.to(Iterator.scala:1336)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Error while executing multi-threaded
tasks
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:536)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.reluBackward(LibMatrixDNN.java:391)
	at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.processReluBackwardInstruction(ConvolutionCPInstruction.java:229)
	at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.proc
essInstruction(ConvolutionCPInstruction.java:316)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
	... 31 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:532)
	... 35 more
Caused by: java.lang.NullPointerException
	at org.apache.sysml.runtime.util.ConvolutionUtils.scalarOperations(ConvolutionUtils.java:150)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:231)
	at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:208)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 more
Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
  at scala.Option.foreach(Option.scala:257)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1925)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1938)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  at org.apache.spark.rdd.RDD.collect(RDD.scala:935)
  at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
  at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
  at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSpark.runJob(RemoteParForSpark.java:77)
  at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParFor(ParForProgramBlock.java:1055)
  at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:647)
  ... 66 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block
generated from statement block between lines 0 and 0 -- Error evaluating instruction: CP°relu_backward°outa3·MATRIX·DOUBLE°_mVar3925·MATRIX·DOUBLE°_mVar3928·MATRIX·DOUBLE°-1
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:316)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:217)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:163)
  at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeSetTask(ParWorker.java:167)
  at org.apache.sysml.runtime.controlprogram.parfor.ParWorker.executeTask(ParWorker.java:136)
  at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:75)
  at org.apache.sysml.runtime.controlprogram.parfor.RemoteParForSparkWorker.call(RemoteParForSparkWorker.java:40)
  at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
  at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$3$1.apply(JavaRDDLike.scala:143)
  at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
  at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
  at scala.collection.Iterator$class.foreach(Iterator.scala:893)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
  at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
  at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
  at scala.collection.AbstractIterator.to(Iterator.scala:1336)
  at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
  at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
  at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
  at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:936)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1951)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
  at org.apache.spark.scheduler.Task.run(Task.scala:99)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
  ... 3 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: Error while executing multi-threaded
tasks
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:536)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.reluBackward(LibMatrixDNN.java:391)
  at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.processReluBackwardInstruction(ConvolutionCPInstruction.java:229)
  at org.apache.sysml.runtime.instructions.cp.ConvolutionCPInstruction.processInstruction(ConvolutionCPInstruction.java:316)
  at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
  ... 31 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException
  at java.util.concurrent.FutureTask.report(FutureTask.java:122)
  at java.util.concurrent.FutureTask.get(FutureTask.java:192)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNN.execute(LibMatrixDNN.java:532)
  ... 35 more
Caused by: java.lang.NullPointerException
  at org.apache.sysml.runtime.util.ConvolutionUtils.scalarOperations(ConvolutionUtils.java:150)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:231)
  at org.apache.sysml.runtime.matrix.data.LibMatrixDNNHelper$ReluBackward.call(LibMatrixDNNHelper.java:208)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  ... 3 more
{code}


> Improve Parfor parallelism for deep learning
> --------------------------------------------
>
>                 Key: SYSTEMML-1774
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1774
>             Project: SystemML
>          Issue Type: Improvement
>          Components: Algorithms, Compiler, ParFor
>    Affects Versions: SystemML 1.0
>            Reporter: Fei Hu
>              Labels: deeplearning
>
> When running the  [distributed MNIST LeNet example | https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml],
each mini-batch could ideally run in parallel without interaction. We try to force {{parfor
(j in 1:parallel_batches)}} at line 137 of {{nn/examples/mnist_lenet_distrib_sgd.dml}} to
be {{parfor (j in 1:parallel_batches, mode=REMOTE_SPARK, opt=CONSTRAINED)}} use {{REMOTE_SPARK}}
mode, but got some errors about {{org.apache.sysml.runtime.DMLRuntimeException: Not supported:
Instructions of type other than CP instructions}} using the mode {{SPARK}}, and the error
{{java.lang.NullPointerException}} using the mode {{HYBRID_SPARK}}. More log information can
be found at the following comments. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message