Mailing-List: contact pig-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
Message-ID: <515898871.1233363659603.JavaMail.jira@brutus>
Date: Fri, 30 Jan 2009 17:00:59 -0800 (PST)
From: "Santhosh Srinivasan (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Subject: [jira] Updated: (PIG-560) UTFDataFormatException (encoded string
 too long) is thrown when storing strings > 65536 bytes (in UTF8 form) using
 BinStorage()
In-Reply-To: <384465971.1229023124452.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/PIG-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Santhosh Srinivasan updated PIG-560:
------------------------------------

    Attachment: PIG-560_1.patch

Incorporated comments from Laukik. Submitting a new patch and modified test case. Running the tests now.

> UTFDataFormatException (encoded string too long) is thrown when storing strings > 65536 bytes (in UTF8 form) using BinStorage()
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-560
>                 URL: https://issues.apache.org/jira/browse/PIG-560
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: types_branch
>            Reporter: Pradeep Kamath
>             Fix For: types_branch
>
>         Attachments: PIG-560.patch, PIG-560_1.patch, utf-limit-patch.diff
>
>
> BinStorage() uses DataOutput.writeUTF() and DataInput.readUTF() Java API to write out Strings as UTF-8 bytes and to read them back. From the Javadoc - "First, the total number of bytes needed to represent all the characters of s is calculated. If this number is larger than 65535, then a UTFDataFormatException  is thrown. " (because the writeUTF() API uses 2 bytes to represent the number of bytes). A way to get around this would be to not use writeUTF()/ReadUTF() and instead hand convert the string to the corresponding UTF-8 byte[]  (using String.getBytes("UTF-8") and then write the length of the byte array as an int - this will allow a size of upto 2^32 (2 raised to 32).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.