Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7BFC12502 for ; Thu, 5 May 2011 17:22:20 +0000 (UTC) Received: (qmail 89714 invoked by uid 500); 5 May 2011 17:22:18 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 89690 invoked by uid 500); 5 May 2011 17:22:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 89682 invoked by uid 99); 5 May 2011 17:22:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:22:18 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,RFC_ABUSE_POST,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of skrolle@gmail.com designates 209.85.218.44 as permitted sender) Received: from [209.85.218.44] (HELO mail-yi0-f44.google.com) (209.85.218.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 May 2011 17:22:11 +0000 Received: by yic13 with SMTP id 13so1030905yic.31 for ; Thu, 05 May 2011 10:21:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=OZtoa1kw/NciXCjFXCedllrAGw+j1VQmV9nHDiyoALs=; b=pSTjpoh4MDvh618c8U+SHAeaEWPJiPm7J6PpcsVRPd3lOtllxTw5Gxss2eCTjb+H5Q vA+yBi2XKmnAa+3a6O1M3CkWklNYXj2HC3BsuwEsTIxHBh+G33TG0NsePrz5wwH62gV4 dNWaGAWGMckAwdUdic+ME+q+HrPXXbP5lT2z0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=KD5W3//CGSTFV2dtxH0lmi11bX+qRDTxfpS5tZcE5SstrHbnSkhMrRx9Y9X6dNMp9X A4NoZKh9Ljyj63/0D2nqjy6hjjHiJY0BS7VHTiiwvcnRv+VxcCQQSCz+oOUnnvm0237Q Dl0TZPALDC/q/nA+mL9FwDJv0BZu8rLQCKjn8= MIME-Version: 1.0 Received: by 10.91.149.5 with SMTP id b5mr2490471ago.91.1304616109645; Thu, 05 May 2011 10:21:49 -0700 (PDT) Received: by 10.90.55.2 with HTTP; Thu, 5 May 2011 10:21:49 -0700 (PDT) In-Reply-To: References: <80052FA0-2D52-4B1A-ABAA-9B04F843EFC4@gmx.net> <5828F3D8-B354-4E1D-B740-E52ADB983D76@thelastpickle.com> Date: Thu, 5 May 2011 19:21:49 +0200 Message-ID: Subject: Re: Unicode key encoding problem when upgrading from 0.6.13 to 0.7.5 From: =?ISO-8859-1?Q?Henrik_Schr=F6der?= To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e64095c2c89f2704a28a9b8e X-Virus-Checked: Checked by ClamAV on apache.org --0016e64095c2c89f2704a28a9b8e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I can't run sstable2json on the datafiles from 0.7, it throws the same "Key= s must be written in ascending order." error as compaction. I can run sstable2json on the 0.6 datafiles, but when I tested that the unicode characters in the keys got completely mangled since it outputs keys in string format, not byte format. I'm using the random partitioner, and as I said before, in my little test I can see the keys using get_range_slices and sstablekeys, but not using get or get_multi_slices. But the actual datafiles with our live data are too corrupted to get anything, sstablekeys throws the error, cleanup throws the error, etc. /Henrik Schr=C3=B6der On Thu, May 5, 2011 at 13:57, aaron morton wrote: > The hard core way to fix the data is export to json with sstable2json, ha= nd > edit, and then json2sstable it back. > > Also to confirm, this only happens when data is written in 0.6 and then > tried to read back in 0.7? > > And you what partitioner are you using ? You can still see the keys ? > > Can you use sstable2json agains tthe 0.6 data ? > > Looking at you last email something looks fishy about the encoding... > " > My two keys that I send in my test program are 0xe695b0e69982e99693 and > 0x666f6f, which decodes to "=E6=95=B0=E6=99=82=E9=96=93" and "foo" respec= tively. > " > > There are 9 bytes encoded there I would expect a multiple of 2 for each > character. (using UTF-16 surrogate pairs > http://en.wikipedia.org/wiki/UTF-16/UCS-2 ) > > I looked the characters up and their encoding is different here > =E6=95=B0 0x6570 http://www.fileformat.info/info/unicode/char/6570/index.= htm > =E6=99=82 0x6642 http://www.fileformat.info/info/unicode/char/6642/index.= htm > =E9=96=93 0x9593 http://www.fileformat.info/info/unicode/char/9593/index.= htm > > Am I missing something ? > > Hope that helps. > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 5 May 2011, at 23:09, Henrik Schr=C3=B6der wrote: > > Yes, the keys were written to 0.6, but when I looked through the thrift > client code for 0.6, it explicitly converts all string keys to UTF8 befor= e > sending them over to the server so the encoding *should* be right, and af= ter > the upgrade to 0.7.5, sstablekeys prints out the correct byte values for > those keys, but Cassandra itself is unable to get those rows. > > I ran some more tests yesterday with a clean database where I only wrote > two rows, one with an ascii key and one with a unicode key, upgraded to > 0.7.5, ran nodetool cleanup, and that actually fixed it. After cleanup, t= he > server could fetch both rows correctly. > > However, when I tried to do the same thing with a snapshot of our live > database where we have ~2 million keys, out of which ~1000 are unicode, > cleanup failed with a lot of "Keys must be written in descending order" > exceptions. I've tried various combinations of cleanup and scrub, running > cleanup before upgrading, etc, but I've yet to find something that fixes = all > the problems without losing those rows. > > > /Henrik > > On Thu, May 5, 2011 at 12:48, aaron morton wrote= : > >> I take it back, the problem started in 0.6 where keys were strings. >> Looking into how 0.6 did it's thing >> >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 5 May 2011, at 22:36, aaron morton wrote: >> >> Interesting but as we are dealing with keys it should not matter as they >> are treated as byte buffers. >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 5 May 2011, at 04:53, Daniel Doubleday wrote: >> >> This is a bit of a wild guess but Windows and encoding and 0.7.5 sounds >> like >> >> https://issues.apache.org/jira/browse/CASSANDRA-2367 >> >> >> On May 3, 2011, at 5:15 PM, Henrik Schr=C3=B6der wrote: >> >> Hey everyone, >> >> We did some tests before upgrading our Cassandra cluster from 0.6 to 0.7= , >> just to make sure that the change in how keys are encoded wouldn't cause= us >> any dataloss. Unfortunately it seems that rows stored under a unicode ke= y >> couldn't be retrieved after the upgrade. We're running everything on >> Windows, and we're using the generated thrift client in C# to access it. >> >> I managed to make a minimal test to reproduce the error consistently: >> >> First, I started up Cassandra 0.6.13 with an empty data directory, and a >> really simple config with a single keyspace with a single bytestype >> columnfamily. >> I wrote two rows, each with a single column with a simple column name an= d >> a 1-byte value of "1". The first row had a key using only ascii chars >> ('foo'), and the second row had a key using unicode chars ('=E3=83=89=E3= =83=A1=E3=82=A4=E3=83=B3=E3=82=A6'). >> >> Using multi_get, and both those keys, I got both columns back, as >> expected. >> Using multi_get_slice and both those keys, I got both columns back, as >> expected. >> I also did a get_range_slices to get all rows in the columnfamily, and I >> got both columns back, as expected. >> >> So far so good. Then I drain and shut down Cassandra 0.6.13, and start u= p >> Cassandra 0.7.5, pointing to the same data directory, with a config >> containing the same keyspace, and I run the schematool import command. >> >> I then start up my test program that uses the new thrift api, and run so= me >> commands. >> >> Using multi_get_slice, and those two keys encoded as UTF8 byte-arrays, I >> only get back one column, the one under the key 'foo'. The other row I >> simply can't retrieve. >> >> However, when I use get_range_slices to get all rows, I get back two row= s, >> with the correct column values, and the byte-array keys are identical to= my >> encoded keys, and when I decode the byte-arrays as UTF8 drings, I get ba= ck >> my two original keys. This means that both my rows are still there, the = keys >> as output by Cassandra are identical to the original string keys I used = when >> I created the rows in 0.6.13, but it's just impossible to retrieve the >> second row. >> >> To continue the test, I inserted a row with the key '=E3=83=89=E3=83=A1= =E3=82=A4=E3=83=B3=E3=82=A6' encoded as >> UTF-8 again, and gave it a similar column as the original, but with a 1-= byte >> value of "2". >> >> Now, when I use multi_get_slice with my two encoded keys, I get back two >> rows, the 'foo' row has the old value as expected, and the other row has= the >> new value as expected. >> >> However, when I use get_range_slices to get all rows, I get back *three* >> rows, two of which have the *exact same* byte-array key, one has the old >> column, one has the new column. >> >> >> How is this possible? How can there be two different rows with the exact >> same key? I'm guessing that it's related to the encoding of string keys = in >> 0.6, and that the internal representation is off somehow. I checked the >> generated thrift client for 0.6, and it UTF8-encodes all keys before sen= ding >> them to the server, so it should be UTF8 all the way, but apparently it >> isn't. >> >> Has anyone else experienced the same problem? Is it a platform-specific >> problem? Is there a way to avoid this and upgrade from 0.6 to 0.7 and no= t >> lose any rows? I would also really like to know which byte-array I shoul= d >> send in to get back that second row, there's gotta be some key that can = be >> used to get it, the row is still there after all. >> >> >> /Henrik Schr=C3=B6der >> >> >> >> >> > > --0016e64095c2c89f2704a28a9b8e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I can't run sstable2json on the datafiles from 0.7, it throws the same = "Keys must be written in ascending order." error as compaction.I can run sstable2json on the 0.6 datafiles, but when I tested that the u= nicode characters in the keys got completely mangled since it outputs keys = in string format, not byte format.

I'm using the random partitioner, and as I said before, in my littl= e test I can see the keys using get_range_slices and sstablekeys, but not u= sing get or get_multi_slices. But the actual datafiles with our live data a= re too corrupted to get anything, sstablekeys throws the error, cleanup thr= ows the error, etc.


/Henrik Schr=C3=B6der

On Thu, May= 5, 2011 at 13:57, aaron morton <aaron@thelastpickle.com> wrote:
The hard core way to fix the data is ex= port to json with sstable2json, hand edit, and then json2sstable it back.= =C2=A0

Also to confirm, this only happens when data is w= ritten in 0.6 and then tried to read back in 0.7?

And you what partitioner are you using ? You can still = see the keys ?

Can you use sstable2json agains tth= e 0.6 data ?

Looking at you last email something l= ooks fishy about the encoding...
"
My two keys that I send in my = test program are 0xe695b0e69982e99693 and 0x666f6f, which decodes to "= =E6=95=B0=E6=99=82=E9=96=93" and "foo" respectively.
"

There are 9 bytes encoded there I would expect a multi= ple of 2 for each character. (using UTF-16 surrogate pairs=C2=A0http://en.wiki= pedia.org/wiki/UTF-16/UCS-2=C2=A0)

I looked the characters up and their encoding is differ= ent here=C2=A0
=E9=96=93 0x9593=C2=A0http://www.fileformat.info/info/unicode/char/9593/index.htm

Am I missing something ?

Hope = that helps.=C2=A0
-----------------
Aaron Morton
Freelance Cassand= ra Developer
@aaronmorton

On 5 May 2011, at 23:= 09, Henrik Schr=C3=B6der wrote:

Yes, the= keys were written to 0.6, but when I looked through the thrift client code= for 0.6, it explicitly converts all string keys to UTF8 before sending the= m over to the server so the encoding *should* be right, and after the upgra= de to 0.7.5, sstablekeys prints out the correct byte values for those keys,= but Cassandra itself is unable to get those rows.

I ran some more tests yesterday with a clean database where I only wrot= e two rows, one with an ascii key and one with a unicode key, upgraded to 0= .7.5, ran nodetool cleanup, and that actually fixed it. After cleanup, the = server could fetch both rows correctly.

However, when I tried to do the same thing with a snapshot of our live = database where we have ~2 million keys, out of which ~1000 are unicode, cle= anup failed with a lot of "Keys must be written in descending order&qu= ot; exceptions. I've tried various combinations of cleanup and scrub, r= unning cleanup before upgrading, etc, but I've yet to find something th= at fixes all the problems without losing those rows.


/Henrik

On Thu, May 5, 2011 at 12= :48, aaron morton <aaron@thelastpickle.com> wrote:
=
I take it back, the problem started in = 0.6 where keys were strings. Looking into how 0.6 did it's thing

-----------------
Aaron Morton
Freelance Cass= andra Developer
@aaronmorton

On 5 May 2011, at 22:36, aaron mor= ton wrote:

Interesting but as we are dealing with keys it should not matter as = they are treated as byte buffers.=C2=A0

-----------------
Aaron Morton
Freelance Cass= andra Developer
@aaronmorton

On 5 May 2011, at 04:53, Daniel Doubleday wrote:

This is a bit of= a wild guess but Windows and encoding and 0.7.5 sounds like

=

=C2=A0
On May 3, 2011, at 5:15 PM, Henrik Schr=C3=B6der wrote:

=
Hey everyone,

We did some tests before upg= rading our Cassandra cluster from 0.6 to 0.7, just to make sure that the ch= ange in how keys are encoded wouldn't cause us any dataloss. Unfortunat= ely it seems that rows stored under a unicode key couldn't be retrieved= after the upgrade. We're running everything on Windows, and we're = using the generated thrift client in C# to access it.

I managed to make a minimal test to reproduce the error consistently:
First, I started up Cassandra 0.6.13 with an empty data directory, an= d a really simple config with a single keyspace with a single bytestype col= umnfamily.
I wrote two rows, each with a single column with a simple column name and a= 1-byte value of "1". The first row had a key using only ascii ch= ars ('foo'), and the second row had a key using unicode chars ('= ;=E3=83=89=E3=83=A1=E3=82=A4=E3=83=B3=E3=82=A6').

Using multi_get, and both those keys, I got both columns back, as expec= ted.
Using multi_get_slice and both those keys, I got both columns back,= as expected.
I also did a get_range_slices to get all rows in the colum= nfamily, and I got both columns back, as expected.

So far so good. Then I drain and shut down Cassandra 0.6.13, and start = up Cassandra 0.7.5, pointing to the same data directory, with a config cont= aining the same keyspace, and I run the schematool import command.

I then start up my test program that uses the new thrift api, and run some = commands.

Using multi_get_slice, and those two keys encoded as UTF8 = byte-arrays, I only get back one column, the one under the key 'foo'= ;. The other row I simply can't retrieve.

However, when I use get_range_slices to get all rows, I get back two ro= ws, with the correct column values, and the byte-array keys are identical t= o my encoded keys, and when I decode the byte-arrays as UTF8 drings, I get = back my two original keys. This means that both my rows are still there, th= e keys as output by Cassandra are identical to the original string keys I u= sed when I created the rows in 0.6.13, but it's just impossible to retr= ieve the second row.

To continue the test, I inserted a row with the key '=E3=83=89=E3= =83=A1=E3=82=A4=E3=83=B3=E3=82=A6' encoded as UTF-8 again, and gave it = a similar column as the original, but with a 1-byte value of "2".=

Now, when I use multi_get_slice with my two encoded keys, I get bac= k two rows, the 'foo' row has the old value as expected, and the ot= her row has the new value as expected.

However, when I use get_range_slices to get all rows, I get back *three= * rows, two of which have the *exact same* byte-array key, one has the old = column, one has the new column.


How is this possible? How can t= here be two different rows with the exact same key? I'm guessing that i= t's related to the encoding of string keys in 0.6, and that the interna= l representation is off somehow. I checked the generated thrift client for = 0.6, and it UTF8-encodes all keys before sending them to the server, so it = should be UTF8 all the way, but apparently it isn't.

Has anyone else experienced the same problem? Is it a platform-specific= problem? Is there a way to avoid this and upgrade from 0.6 to 0.7 and not = lose any rows? I would also really like to know which byte-array I should s= end in to get back that second row, there's gotta be some key that can = be used to get it, the row is still there after all.


/Henrik Schr=C3=B6der






--0016e64095c2c89f2704a28a9b8e--