flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Richter <s.rich...@data-artisans.com>
Subject Re: FlinkML ALS matrix factorization: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to an exception: null
Date Wed, 31 Aug 2016 15:57:56 GMT
Hi,

could you provide the log outputs for your job (ideally with debug logging enabled)?

Best,
Stefan

> Am 31.08.2016 um 14:40 schrieb ANDREA SPINA <74598@studenti.unimore.it>:
> 
> Hi everyone.
> I'm running the FlinkML ALS matrix factorization and I bumped into the following exception:
> 
> org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
Job execution failed.
> 	at org.apache.flink.client.program.Client.runBlocking(Client.java:381)
> 	at org.apache.flink.client.program.Client.runBlocking(Client.java:355)
> 	at org.apache.flink.client.program.Client.runBlocking(Client.java:315)
> 	at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:60)
> 	at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:652)
> 	at org.apache.flink.ml.common.FlinkMLTools$.persist(FlinkMLTools.scala:94)
> 	at org.apache.flink.ml.recommendation.ALS$$anon$115.fit(ALS.scala:507)
> 	at org.apache.flink.ml.recommendation.ALS$$anon$115.fit(ALS.scala:433)
> 	at org.apache.flink.ml.pipeline.Estimator$class.fit(Estimator.scala:55)
> 	at org.apache.flink.ml.recommendation.ALS.fit(ALS.scala:122)
> 	at dima.tu.berlin.benchmark.flink.als.RUN$.main(RUN.scala:78)
> 	at dima.tu.berlin.benchmark.flink.als.RUN.main(RUN.scala)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
> 	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
> 	at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
> 	at org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:866)
> 	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:333)
> 	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1192)
> 	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1243)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:717)
> 	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:663)
> 	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:663)
> 	at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> 	at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> 	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> 	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
> 	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> 	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> 	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> 	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> 	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.RuntimeException: Initializing the input processing failed: Error
obtaining the sorted input: Thread 'SortMerger Reading Thread' terminated due to an exception:
null
> 	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:325)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger
Reading Thread' terminated due to an exception: null
> 	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:619)
> 	at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1079)
> 	at org.apache.flink.runtime.operators.BatchTask.initLocalStrategies(BatchTask.java:819)
> 	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:321)
> 	... 2 more
> Caused by: java.io.IOException: Thread 'SortMerger Reading Thread' terminated due to
an exception: null
> 	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
> Caused by: org.apache.flink.runtime.io.network.partition.ProducerFailedException
> 	at org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.getNextLookAhead(LocalInputChannel.java:270)
> 	at org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.onNotification(LocalInputChannel.java:238)
> 	at org.apache.flink.runtime.io.network.partition.PipelinedSubpartition.release(PipelinedSubpartition.java:158)
> 	at org.apache.flink.runtime.io.network.partition.ResultPartition.release(ResultPartition.java:320)
> 	at org.apache.flink.runtime.io.network.partition.ResultPartitionManager.releasePartitionsProducedBy(ResultPartitionManager.java:95)
> 	at org.apache.flink.runtime.io.network.NetworkEnvironment.unregisterTask(NetworkEnvironment.java:370)
> 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:657)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> I'm running with flink-1.0.3. I really can't figure out the reason behind that.
> 
> My code simply calls the library as follows:
> 
> val als = ALS()
>   .setIterations(numIterations)
>   .setNumFactors(rank)
>   .setBlocks(degreeOfParallelism)
>   .setSeed(42)
>   .setTemporaryPath(tempPath)
> 
> als.fit(ratings, parameters)
> 
> val (users, items) = als.factorsOption match {
>   case Some(_) => als.factorsOption.get
>   case _ => throw new RuntimeException
> }
> 
> users.writeAsText(outputPath, WriteMode.OVERWRITE)
> items.writeAsText(outputPath, WriteMode.OVERWRITE)
> 
> env.execute("ALS matrix factorization")
> 
> where 
> - ratings as the input dataset contains (uid, iid, rate) rows about 8e6 users, 1e6 items
and 700 rating per user average.
> - numIterations 10
> - rank 50
> - degreeOfParallelism 240
> 
> The error seems to be related to the final .persists() call.
> at org.apache.flink.ml.recommendation.ALS$$anon$115.fit(ALS.scala:507)
> 
> I'm running with a 15 nodes cluster - 16cpus per node - with the following valuable properties:
> 
> jobmanager.heap.mb = 2048
> taskmanager.memory.fraction = 0.5
> taskmanager.heap.mb = 28672
> taskmanager.network.bufferSizeInBytes = 32768
> taskmanager.network.numberOfBuffers = 98304
> akka.ask.timeout = 300s
> 
> Any help will be appreciated. Thank you.
> 
> -- 
> Andrea Spina
> N.Tessera: 74598
> MAT: 89369
> Ingegneria Informatica [LM] (D.M. 270)


Mime
View raw message