Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 9 Apr 2013 22:26:16 +0000 (UTC)
From: "Phabricator (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12637603.1363631683459.138975.1365546376619@arcas>
In-Reply-To: <JIRA.12637603.1363631683459@arcas>
References: <JIRA.12637603.1363631683459@arcas>
Subject: [jira] [Commented] (HIVE-4199) ORC writer doesn't handle non-UTF8
 encoded Text properly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627200#comment-13627200 ] 

Phabricator commented on HIVE-4199:
-----------------------------------

sxyuan has commented on the revision "HIVE-4199 [jira] ORC writer doesn't handle non-UTF8 encoded Text properly".

  Inline comments.

INLINE COMMENTS
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/StringRedBlackTree.java:44 The reason why I kept the add(String) method is that it can avoid doing two copies when the original data is actually a String. If the dictionary only takes Text objects, the writer will have to convert the String to a new Text object, and then set(Text) will copy the bytes over to the dictionary's internal Text object.
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java:717 I've looked into adding statistics for non-UTF8 strings, but I discovered that the stats are serialized to Protobuf objects which require strings to be UTF8 encoded. Do you have any suggestions?

REVISION DETAIL
  https://reviews.facebook.net/D9501

To: kevinwilfong, sxyuan
Cc: JIRA, omalley

                
> ORC writer doesn't handle non-UTF8 encoded Text properly
> --------------------------------------------------------
>
>                 Key: HIVE-4199
>                 URL: https://issues.apache.org/jira/browse/HIVE-4199
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>            Reporter: Samuel Yuan
>            Assignee: Samuel Yuan
>            Priority: Minor
>         Attachments: HIVE-4199.HIVE-4199.HIVE-4199.D9501.1.patch, HIVE-4199.HIVE-4199.HIVE-4199.D9501.2.patch, HIVE-4199.HIVE-4199.HIVE-4199.D9501.3.patch, HIVE-4199.HIVE-4199.HIVE-4199.D9501.4.patch
>
>
> StringTreeWriter currently converts fields stored as Text objects into Strings. This can lose information (see http://en.wikipedia.org/wiki/Replacement_character#Replacement_character), and is also unnecessary since the dictionary stores Text objects.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira