flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1208) Skip comment lines in CSV input format. Allow user to specify comment character.
Date Mon, 24 Nov 2014 22:03:14 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223607#comment-14223607

ASF GitHub Bot commented on FLINK-1208:

Github user fhueske commented on a diff in the pull request:

    --- Diff: flink-java/src/main/java/org/apache/flink/api/java/io/CsvInputFormat.java ---
    @@ -137,6 +239,7 @@ public OUT readRecord(OUT reuse, byte[] bytes, int offset, int numBytes)
     			return reuse;
     		} else {
    +			this.invalidLineCount++;
    --- End diff --
    The DelimitedIF forwards the returned `null` to the DataSourceTask where it is ignored.
Hence, there is no checking for input correctness. If the CsvIF was configured with lenient
= false, we need to raise an exception here instead of returning null.

> Skip comment lines in CSV input format. Allow user to specify comment character.
> --------------------------------------------------------------------------------
>                 Key: FLINK-1208
>                 URL: https://issues.apache.org/jira/browse/FLINK-1208
>             Project: Flink
>          Issue Type: Improvement
>          Components: Java API, Scala API
>    Affects Versions: 0.8-incubating
>            Reporter: Aljoscha Krettek
>            Assignee: Felix Neutatz
>            Priority: Minor
>              Labels: starter
> The current skipFirstLine is limited. Skipping arbitrary lines that start with a certain
character would be much more flexible while still easy to implement.

This message was sent by Atlassian JIRA

View raw message