flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: An error occurred in the channel (The channel is erroneous.)
Date Wed, 08 Oct 2014 19:14:00 GMT
Hey Florian,

sorry that you run into these annoying problems, but in this case I think it is pretty much
what it says and the system is running out of space.

The asynchronous I/O writer thread tries to write to a file channel, which throws the corresponding
IOException "No space left on device".

I would suggest that you manually	clean up the tmp directories of your machine(s). If you
didn't configure anything, Flink falls back to "java.io.tmpdir", which is the system dependent
temp file path, e.g. /tmp.

Please keep us updated if this helps or not.

(Flink should actually clean up everything when the disk based operators close. I'm not sure
what happens when a program dies with an exception.)

PS: We should check whether we can improve the error message or at least log the used temp
directories when the IOManager starts. I will open a corresponding issue.

On 08 Oct 2014, at 20:49, Florian Hönicke <rockstarflo@gmail.com> wrote:

> Thanks allot.
> I switched to flink0.6.1
> Now i get a slightly different exception:  "The channel is erroneous."
> 
> ...
> ...rojectFlatJoinFunction) (159/162) switched to CANCELED
> 10/08/2014 20:27:58: Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction)
(160/162) switched to CANCELED
> 10/08/2014 20:27:58: Join(org.apache.flink.api.java.operators.JoinOperator$ProjectFlatJoinFunction)
(161/162) switched to CANCELED
> 10/08/2014 20:27:58:    Job execution switched to status FAILED
> Error: The program execution failed: java.lang.Exception: The data preparation for task
'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)' , caused an error:
Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated due to an
exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
>        at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
>        at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>        at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger
spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
>        at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
>        at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
>        ... 3 more
> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to
an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
> Caused by: java.io.IOException: The channel is erroneous.
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
>        at org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
>        at org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
>        at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
>        at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
> Caused by: java.io.IOException: No space left on device
>        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>        at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>        at org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
>        at org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)
> 
> org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
java.lang.Exception: The data preparation for task 'Reduce(org.apache.flink.allIn.StratosphereMultiFlink6Job$ReduceSim2)'
, caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread' terminated
due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:485)
>        at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:375)
>        at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:265)
>        at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Thread 'SortMerger
spilling thread' terminated due to an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:616)
>        at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1131)
>        at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:91)
>        at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:480)
>        ... 3 more
> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' terminated due to
an exception: The channel is erroneous.
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:801)
> Caused by: java.io.IOException: The channel is erroneous.
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.checkErroneous(ChannelAccess.java:132)
>        at org.apache.flink.runtime.io.disk.iomanager.BlockChannelWriter.writeBlock(BlockChannelWriter.java:73)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.writeSegment(ChannelWriterOutputView.java:220)
>        at org.apache.flink.runtime.io.disk.iomanager.ChannelWriterOutputView.nextSegment(ChannelWriterOutputView.java:206)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.advance(AbstractPagedOutputView.java:140)
>        at org.apache.flink.runtime.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:267)
>        at org.apache.flink.api.common.typeutils.base.IntSerializer.copy(IntSerializer.java:74)
>        at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(TupleSerializer.java:124)
>        at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1316)
>        at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:798)
> Caused by: java.io.IOException: No space left on device
>        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>        at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>        at org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(BlockChannelAccess.java:259)
>        at org.apache.flink.runtime.io.disk.iomanager.IOManager$WriterThread.run(IOManager.java:644)
> 
>        at org.apache.flink.client.program.Client.run(Client.java:325)
>        at org.apache.flink.client.program.Client.run(Client.java:291)
>        at org.apache.flink.client.program.Client.run(Client.java:285)
>        at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:54)
>        at org.apache.flink.allIn.StratosphereMultiFlink6Job.main(StratosphereMultiFlink6Job.java:72)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:389)
>        at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:307)
>        at org.apache.flink.client.program.Client.run(Client.java:244)
>        at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:332)
>        at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
>        at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:930)
>        at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954)
> 
> 
> Am 08.10.2014 15:04, schrieb Ufuk Celebi:
>> Hey Florian,
>> 
>> this is a known issue and the commit, which introduced the problem has been reverted
for the 0.6.1 release. Could you please work with that version?
>> 
>> – Ufuk
>> 
>> On 08 Oct 2014, at 14:34, Florian Hönicke <rockstarflo@gmail.com> wrote:
>> 
>>> Hi,
>>> 
>>> I get a runtime error while executing my flink0.6 job.
>>> "An error occurred in the channel"
>>> Could anyone help me?
>>> 
>>> Greetings
>>> Florian
>>> <out.txt>
> 
> <StratosphereMultiFlink6Job.java><flink-conf.yaml>


Mime
View raw message