hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-63) PigStorage does not properly handle UTF8 data
Date Fri, 18 Jan 2008 21:06:33 GMT

    [ https://issues.apache.org/jira/browse/PIG-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560544#action_12560544

Alan Gates commented on PIG-63:

When I apply this patch and run the tests, they fail with the following error message:

08/01/18 11:50:16 ERROR apache.pig: Error message from task (map) tip_200801071337_2639_m_000000
	at org.apache.pig.impl.io.BufferedPositionedInputStream.readLine(BufferedPositionedInputStream.java:146)
	at org.apache.pig.builtin.PigStorage.getNext(PigStorage.java:62)
	at org.apache.pig.impl.mapreduceExec.PigInputFormat$PigRecordReader.next(PigInputFormat.java:188)
	at org.apache.pig.impl.mapreduceExec.PigInputFormat$PigRecordReader.next(PigInputFormat.java:131)
	at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:174)
	at org.apache.pig.impl.mapreduceExec.PigMapReduce.run(PigMapReduce.java:114)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

> PigStorage does not properly handle UTF8 data
> ---------------------------------------------
>                 Key: PIG-63
>                 URL: https://issues.apache.org/jira/browse/PIG-63
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Olga Natkovich
>            Assignee: Benjamin Reed
>         Attachments: utf8.patch, utf8.patch
> From Ben:
> I just checked the code and the problem seems to be PigStorage. getNext() uses
> readLine() which does not handle UTF8 correctly. putNext() also uses default encoder
rather than UTF8 explicitly.
> Internally and in BinStorage UTF8 appears to be handled correctly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message