Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Fri, 28 Apr 2017 16:34:04 +0000 (UTC)
From: "Davor Bonaci (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13067665.1493379986000.80349.1493397244071@Atlassian.JIRA>
In-Reply-To: <JIRA.13067665.1493379986000@Atlassian.JIRA>
References: <JIRA.13067665.1493379986000@Atlassian.JIRA> <JIRA.13067665.1493379986880@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-2110) HDFSFileSource throws
 IndexOutOfBoundsException when trying to read big file (gaming_data1.csv)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 28 Apr 2017 16:34:12 -0000


    [ https://issues.apache.org/jira/browse/BEAM-2110?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15989=
117#comment-15989117 ]=20

Davor Bonaci commented on BEAM-2110:
------------------------------------

[~jbonofre], any chance you'd be able to help with this exception?

> HDFSFileSource throws IndexOutOfBoundsException when trying to read big f=
ile (gaming_data1.csv)
> -------------------------------------------------------------------------=
----------------------
>
>                 Key: BEAM-2110
>                 URL: https://issues.apache.org/jira/browse/BEAM-2110
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink, sdk-java-extensions
>         Environment: CentOS 7, Oracle JDK 8
>            Reporter: Gergely Nov=C3=A1k
>            Assignee: Jean-Baptiste Onofr=C3=A9
>
> I modified the wordcount example to read from HDFS with this code:
> {code}
> pipeline.apply(Read.from(HDFSFileSource.fromText(options.getInput())))
> {code}
> This worked for a number of small files I tried with. But with the includ=
ed example: gs://apache-beam-samples/game/gaming_data*.csv (moved to HDFS) =
fails with the following trace:
> {noformat}
> Caused by: java.lang.IndexOutOfBoundsException
> =09at java.nio.Buffer.checkBounds(Buffer.java:567)
> =09at java.nio.ByteBuffer.get(ByteBuffer.java:686)
> =09at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:285)
> =09at org.apache.hadoop.hdfs.RemoteBlockReader2.read(RemoteBlockReader2.j=
ava:168)
> =09at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSI=
nputStream.java:775)
> =09at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.jav=
a:831)
> =09at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStre=
am.java:891)
> =09at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
> =09at java.io.DataInputStream.read(DataInputStream.java:149)
> =09at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.f=
illBuffer(UncompressedSplitLineReader.java:59)
> =09at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:2=
16)
> =09at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
> =09at org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.r=
eadLine(UncompressedSplitLineReader.java:91)
> =09at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteO=
rderMark(LineRecordReader.java:144)
> =09at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue=
(LineRecordReader.java:184)
> =09at org.apache.beam.sdk.io.hdfs.HDFSFileSource$HDFSFileReader.advance(H=
DFSFileSource.java:492)
> =09at org.apache.beam.sdk.io.hdfs.HDFSFileSource$HDFSFileReader.start(HDF=
SFileSource.java:465)
> =09at org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeSt=
art(ReaderInvocationUtil.java:50)
> =09at org.apache.beam.runners.flink.translation.wrappers.SourceInputForma=
t.open(SourceInputFormat.java:79)
> =09at org.apache.beam.runners.flink.translation.wrappers.SourceInputForma=
t.open(SourceInputFormat.java:45)
> =09at org.apache.flink.runtime.operators.DataSourceTask.invoke(DataSource=
Task.java:144)
> =09at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
> =09at java.lang.Thread.run(Thread.java:745)
> {noformat}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)