flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Exception when running WC
Date Wed, 10 Sep 2014 09:28:04 GMT
Looks like a spurious I/O failure during a file read, outside of our code.

I agree with Robert, this looks like a rare OS/disk/java error, and is not
for us to worry about.

On Wed, Sep 10, 2014 at 10:57 AM, Robert Metzger <rmetzger@apache.org>
wrote:

> The error message is not very specific. I was unable to find any
> explanation on this in the web.
>
> I would ignore the message for now. I think its not related to the python
> interface. I'm running a lot of experiments on the cluster and I have not
> seen this error there before.
> If we see the issue appearing more frequently, we should start looking into
> patterns (which machine?, harddisk?, only a specific job, etc.).
>
>
> On Wed, Sep 10, 2014 at 2:57 AM, Chesnay Schepler <
> chesnay.schepler@fu-berlin.de> wrote:
>
> > Hello,
> >
> > tonight i was running a WordCount job with the Python API, and halfway
> > through i got the exception below.
> > the issue did not occur again after ressubmitting the job.
> > DOP=160
> > taskslots=8
> > filesize=100GB
> >
> >    org.apache.flink.client.program.ProgramInvocationException: The
> >    program execution failed: java.lang.RuntimeException: An error
> >    occurred while reading the next record: The channel is erroneous.
> >         at
> >    org.apache.flink.runtime.util.KeyGroupedIterator$
> > ValuesIterator.hasNext(KeyGroupedIterator.java:202)
> >         at
> >
> org.apache.flink.languagebinding.api.java.streaming.Sender.sendRecords(
> > Sender.java:57)
> >         at
> >    org.apache.flink.languagebinding.api.java.streaming.Streamer.stream(
> > Streamer.java:106)
> >         at
> >    org.apache.flink.languagebinding.api.java.python.functions.
> > PythonGroupReduce.reduce(PythonGroupReduce.java:77)
> >         at
> >    org.apache.flink.runtime.operators.GroupReduceDriver.
> > run(GroupReduceDriver.java:108)
> >         at
> >    org.apache.flink.runtime.operators.RegularPactTask.run(
> > RegularPactTask.java:509)
> >         at
> >    org.apache.flink.runtime.operators.RegularPactTask.
> > invoke(RegularPactTask.java:374)
> >         at
> >    org.apache.flink.runtime.execution.RuntimeEnvironment.
> > run(RuntimeEnvironment.java:265)
> >         at java.lang.Thread.run(Thread.java:745)
> >    Caused by: java.io.IOException: The channel is erroneous.
> >         at
> >    org.apache.flink.runtime.io.disk.iomanager.ChannelAccess.
> > checkErroneous(ChannelAccess.java:132)
> >         at
> >    org.apache.flink.runtime.io.disk.iomanager.
> > BlockChannelReader.readBlock(BlockChannelReader.java:75)
> >         at
> >    org.apache.flink.runtime.io.disk.iomanager.ChannelReaderInputView.
> > sendReadRequest(ChannelReaderInputView.java:263)
> >         at
> >    org.apache.flink.runtime.io.disk.iomanager.ChannelReaderInputView.
> > nextSegment(ChannelReaderInputView.java:226)
> >         at
> >    org.apache.flink.runtime.memorymanager.AbstractPagedInputView.advance(
> > AbstractPagedInputView.java:159)
> >         at
> >
> org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readByte(
> > AbstractPagedInputView.java:270)
> >         at
> >    org.apache.flink.runtime.memorymanager.AbstractPagedInputView.
> > readUnsignedByte(AbstractPagedInputView.java:277)
> >         at
> >    org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readInt(
> > AbstractPagedInputView.java:340)
> >         at
> >    org.apache.flink.api.common.typeutils.base.IntSerializer.
> > deserialize(IntSerializer.java:69)
> >         at
> >    org.apache.flink.api.common.typeutils.base.IntSerializer.
> > deserialize(IntSerializer.java:28)
> >         at
> >    org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.deserialize(TupleSerializer.java:115)
> >         at
> >    org.apache.flink.api.java.typeutils.runtime.
> > TupleSerializer.deserialize(TupleSerializer.java:30)
> >         at
> >    org.apache.flink.runtime.io.disk.ChannelReaderInputViewIterator.next(
> > ChannelReaderInputViewIterator.java:86)
> >         at
> >    org.apache.flink.runtime.operators.sort.MergeIterator$
> > HeadStream.nextHead(MergeIterator.java:131)
> >         at
> >    org.apache.flink.runtime.operators.sort.MergeIterator.
> > next(MergeIterator.java:89)
> >         at
> >    org.apache.flink.runtime.util.KeyGroupedIterator$
> > ValuesIterator.hasNext(KeyGroupedIterator.java:177)
> >         ... 8 more
> >    Caused by: java.io.IOException: Input/output error
> >         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> >         at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
> >         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> >         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> >         at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:149)
> >         at
> >    org.apache.flink.runtime.io.disk.iomanager.SegmentReadRequest.read(
> > BlockChannelAccess.java:221)
> >         at
> >    org.apache.flink.runtime.io.disk.iomanager.IOManager$
> > ReaderThread.run(IOManager.java:551)
> >         at org.apache.flink.client.program.Client.run(Client.java:321)
> >         at org.apache.flink.client.program.Client.run(Client.java:287)
> >         at org.apache.flink.client.program.Client.run(Client.java:281)
> >         at
> >    org.apache.flink.client.program.ContextEnvironment.
> > execute(ContextEnvironment.java:54)
> >         at
> >    org.apache.flink.api.java.ExecutionEnvironment.execute(
> > ExecutionEnvironment.java:519)
> >         at
> >    org.apache.flink.languagebinding.api.java.python.PythonExecutor.main(
> > PythonExecutor.java:119)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at
> >    sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:57)
> >         at
> >    sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at
> >    org.apache.flink.client.program.PackagedProgram.callMainMethod(
> > PackagedProgram.java:389)
> >         at
> >    org.apache.flink.client.program.PackagedProgram.
> > invokeInteractiveModeForExecution(PackagedProgram.java:307)
> >         at org.apache.flink.client.program.Client.run(Client.java:240)
> >         at
> >    org.apache.flink.client.CliFrontend.executeProgram(
> > CliFrontend.java:332)
> >         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
> >         at
> >    org.apache.flink.client.CliFrontend.parseParameters(
> > CliFrontend.java:930)
> >         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954)
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message