flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3921) StringParser not specifying encoding to use
Date Mon, 05 Dec 2016 20:11:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723245#comment-15723245
] 

ASF GitHub Bot commented on FLINK-3921:
---------------------------------------

Github user fhueske commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2901#discussion_r90944589
  
    --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java
---
    @@ -182,8 +186,37 @@ protected DelimitedInputFormat(Path filePath, Configuration configuration)
{
     		}
     		loadConfigParameters(configuration);
     	}
    -	
    -	
    +
    +	/**
    +	 * Get the character set used for the row delimiter, field delimiter,
    +	 * comment string, and {@link FieldParser}.
    +	 *
    +	 * @return the charset
    +	 */
    +	@PublicEvolving
    +	public Charset getCharset() {
    +		if (this.charset == null) {
    +			this.charset = Charset.forName(charsetName);
    +		}
    +		return this.charset;
    +	}
    +
    +	/**
    +	 * Set the name of the character set used for the row delimiter, field
    +	 * delimiter, comment string, and {@link FieldParser}.
    +	 *
    +	 * The delimeters and comment string are interpreted when set. Each
    --- End diff --
    
    `GenericCsvInputFormat` would need to override this method and call `super.setCharset()`
to propagate the change.
    
    What do you think @greghogan?


> StringParser not specifying encoding to use
> -------------------------------------------
>
>                 Key: FLINK-3921
>                 URL: https://issues.apache.org/jira/browse/FLINK-3921
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.0.3
>            Reporter: Tatu Saloranta
>            Assignee: Rekha Joshi
>            Priority: Trivial
>
> Class `flink.types.parser.StringParser` has javadocs indicating that contents are expected
to be Ascii, similar to `StringValueParser`. That makes sense, but when constructing actual
instance, no encoding is specified; on line 66 f.ex:
>    this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents really are always
Ascii (would not count on that as parser is used from CSV reader), not a big deal, but it
can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be used: javadocs
claim ascii, so could be "us-ascii", but could well be UTF-8 or even ISO-8859-1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message