Mailing-List: contact hive-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hive-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of zshao9@gmail.com designates
 209.85.210.185 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=x6BDWT6Ebyn9G7gLsDlEQxSi1SQdnfsBiLMfVTruMlLx81BqjQVQZvsuuLJXJr1mAg
         9DYyNDdCG0yrBVZXrZE7J+DuFnCUlPOh8Y/SV+Sxx+yBWnF7g8lTsxxXGLJNNcobTRzB
         jtCauIpntnz5dhL1dCSWvSwUGiqgAig80qNjY=
MIME-Version: 1.0
In-Reply-To: <bcaf338a0907072359w35b1c500k9aaad65c095b3965@mail.gmail.com>
References: <bcaf338a0907072359w35b1c500k9aaad65c095b3965@mail.gmail.com>
Date: Wed, 8 Jul 2009 00:03:03 -0700
Message-ID: <34fd060d0907080003n42ee211fhc11ef3b01d23c2bc@mail.gmail.com>
Subject: Re: unicode supporting in hive
From: Zheng Shao <zshao9@gmail.com>
To: hive-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Min,

The separators used in Hive are by default ^A, ^B, ^C ... (ascii code
1, 2, 3, etc).
These won't appear in either UTF-8 or GBK:

Please see these code maps for details:
http://en.wikipedia.org/wiki/UTF-8
http://en.wikipedia.org/wiki/GBK


Zheng

On Tue, Jul 7, 2009 at 11:59 PM, Min Zhou<coderplay@gmail.com> wrote:
> Hi all,
> It seems that hive would go wrong when storing unicode strings. Hive use
> byte comparision for delimiting fields of a record(
> see=A0 LazyStruct.java:92, a parse method).
> If we use gbk or utf-8 encoding where characters would need more than 1
> byte, might 2-3 bytes,=A0 then it would by coincidence seperator for
> delimiting fields equal one of byte in our gbk/utf-8 encoding character.
> thus things go wrong.
> Can hive solve the problem above?
>
> Thanks,
> Min
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>


--=20
Yours,
Zheng