Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of ivarley@salesforce.com
 designates 64.18.3.86 as permitted sender)
From: Ian Varley <ivarley@salesforce.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Date: Fri, 8 Jun 2012 10:40:59 -0700
Subject: Re: Collation order of items
Thread-Topic: Collation order of items
Thread-Index: Ac1FneoV+ZJNJTCETquQKGleReo2CA==
Message-ID: <A61D5F42-F0C4-4AED-BC18-751CE9FF7A37@salesforce.com>
References: 
 <CAHtRq2cWFzZifpRwZ7A0rLL0yVeogp4PfR2UcO4zu3FxX+pobg@mail.gmail.com>
 <CADcMMgGwY9DxLDE0hdee3qVSDtVAUKO8-MZoup4nK9vOX+xaaw@mail.gmail.com>
 <CAHtRq2fFFvZcT63-dwUttSWpTP4WNLJopeO2KgCX_i+Ja01Htg@mail.gmail.com>
In-Reply-To: 
 <CAHtRq2fFFvZcT63-dwUttSWpTP4WNLJopeO2KgCX_i+Ja01Htg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Tom, another approach you could take would be to store an ASCII encoded ver=
sion of the string as the row key or column qualifier, and then the full UT=
F-8 string elsewhere (e.g. in the cell value, or even later in the row key)=
. That wouldn't work out the fine sorting (whether "=E8" sorts before or af=
ter "e") but it would solve the gross sorting ("=E8" would always come befo=
re "f"). If you need true UTF-8 collation in the results, you could then im=
plement it as a layer on top of that (in your app, or maybe a co-processor,=
 I'm not sure about the latter). But at least with this approach, you'd be =
able to take advantage of rowkey ranges in your scans, which would probably=
 make up for any time spent doing a secondary sort.

Ian

On Jun 8, 2012, at 12:34 PM, Tom Brown wrote:

> Storing the bytes as native UTF-16 or UTF-32 will not help.  Even
> strings in UTF-8 format can be sorted by their code points when stored
> as bytes.  Unfortunately, that's not really useful for collation as
> characters like "=E8" (U+00E8) should appear between "e" (U+0065) and
> "f" (U+0066), but the code points to not allow this.
>=20
> Thanks anyway!
>=20
> --Tom
>=20
> On Fri, Jun 8, 2012 at 11:14 AM, Stack <stack@duboce.net> wrote:
>> On Fri, Jun 8, 2012 at 9:35 AM, Tom Brown <tombrown52@gmail.com> wrote:
>>> Is there any way to control introduce a different ordering scheme from
>>> the base comparable bytes?  My use case is that I am using UTF-8 data
>>> for my keys, and I would like to have scans use UTF-8 collation.
>>>=20
>>> Could this be done by providing an alternate implementation of
>>> WritableComparable<ImmutableBytesWritable>?
>>>=20
>>> Thanks in advance!
>>>=20
>>=20
>> Unfortunately no Tom.  The database is all sorted the same way.
>> Different sorts per table would complicate system interactions (the
>> catalog tables would have to change sort by table).  It might be
>> doable but it would take some work.
>>=20
>> Can you store your data UTF-16 or UTF-32?  Its a while since I dealt
>> w/ this stuff but IIRC, their sort order is byte order?  (WARNING!  I
>> could be way off here).
>>=20
>> St.Ack