flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3921) StringParser not specifying encoding to use
Date Tue, 20 Sep 2016 13:06:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15506458#comment-15506458

ASF GitHub Bot commented on FLINK-3921:

Github user greghogan commented on the issue:

    Apologies for the long delay. I'd like to attempt to summarize this ticket and pull request
to validate my understanding.
    Previously StringParser was using the system encoding and `GenericCsvInputFormat` was
using UTF-8 for the delimiter and an overloadable UTF-8 for the comment prefix.
    StringParser's quoteCharacter remains a `byte` with no encoding.
    Now GenericCsvInputFormat can be configured with a charset which is used for the delimiter,
comment prefix, and field parsers (only used in StringParser).
    Should `setCommentPrefix(String commentPrefix, Charset charset)` and `setCommentPrefix(String
commentPrefix, String charsetName)` be removed from `GenericCsvInputFormat`? Would different
encodings be used on the same file?
    Allow the user to set the character encoding in `CsvReader` which would be applied in
    Are the new tests checking the encoding? The test strings are using using characters common
to UTF-8 and ASCII. We could instead use one of the UTF-16 encodings from https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html

> StringParser not specifying encoding to use
> -------------------------------------------
>                 Key: FLINK-3921
>                 URL: https://issues.apache.org/jira/browse/FLINK-3921
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.0.3
>            Reporter: Tatu Saloranta
>            Assignee: Rekha Joshi
>            Priority: Trivial
> Class `flink.types.parser.StringParser` has javadocs indicating that contents are expected
to be Ascii, similar to `StringValueParser`. That makes sense, but when constructing actual
instance, no encoding is specified; on line 66 f.ex:
>    this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents really are always
Ascii (would not count on that as parser is used from CSV reader), not a big deal, but it
can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be used: javadocs
claim ascii, so could be "us-ascii", but could well be UTF-8 or even ISO-8859-1.

This message was sent by Atlassian JIRA

View raw message