Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 51B3E10990 for ; Fri, 7 Mar 2014 20:55:10 +0000 (UTC) Received: (qmail 26991 invoked by uid 500); 7 Mar 2014 20:55:06 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 26939 invoked by uid 500); 7 Mar 2014 20:55:05 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 26680 invoked by uid 99); 7 Mar 2014 20:54:59 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Mar 2014 20:54:59 +0000 Date: Fri, 7 Mar 2014 20:54:59 +0000 (UTC) From: "Miles Shang (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-6285) 2.0 HSHA server introduces corrupt data MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-6285?page=3Dcom.atlas= sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D= 13924330#comment-13924330 ]=20 Miles Shang commented on CASSANDRA-6285: ---------------------------------------- To add to [~rbranson]'s input, we're also seeing the same stacktrace as [~m= shuler] (TimeUUID MarshalException). I inspected the row mutations that cau= sed it. Three ranges were nonsensical: the key, the column name, and the va= lue. By nonsensical, I mean that they don't match my expectation of what we= are inserting in production data. All other ranges seemed fine (timestamps= , masks, sizes, cfid). The key, column name, and value were read successful= ly, so their length metadata was good. For our data, the column comparator = is TimeUUID. Our client library is pycassa. Whereas pycassa generates tuuid= s like this: 913d7fea-a631-11e3-8080-808080808080, the nonsensical column n= ames look like this: 22050aa4-de11-e380-8080-80808080800b and this: 10c326e= b-86a4-e211-e380-808080808080. Most are of the first form. By shifting thes= e nonsensical tuuids to the left or right by 2 octets, you get a reasonable= tuuid. I don't have a similar insight into the nonsensical keys and values= , but they could also be left or right shifted. > 2.0 HSHA server introduces corrupt data > --------------------------------------- > > Key: CASSANDRA-6285 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6285 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 4 nodes, shortly updated from 1.2.11 to 2.0.2 > Reporter: David Sauer > Assignee: Pavel Yaskevich > Priority: Critical > Fix For: 2.0.6 > > Attachments: 6285_testnotes1.txt, CASSANDRA-6285-disruptor-heap.p= atch, compaction_test.py > > > After altering everything to LCS the table OpsCenter.rollups60 amd one ot= her none OpsCenter-Table got stuck with everything hanging around in L0. > The compaction started and ran until the logs showed this: > ERROR [CompactionExecutor:111] 2013-11-01 19:14:53,865 CassandraDaemon.ja= va (line 187) Exception in thread Thread[CompactionExecutor:111,1,RMI Runti= me] > java.lang.RuntimeException: Last written key DecoratedKey(132628385146342= 0237, 37382e34362e3132382e3139382d6a7576616c69735f6e6f72785f696e6465785f323= 031335f31305f30382d63616368655f646f63756d656e74736c6f6f6b75702d676574426c6f= 6f6d46696c746572537061636555736564) >=3D current key DecoratedKey(954210699= 457429663, 37382e34362e3132382e3139382d6a7576616c69735f6e6f72785f696e646578= 5f323031335f31305f30382d63616368655f646f63756d656e74736c6f6f6b75702d6765745= 46f74616c4469736b5370616365557365640b0f) writing into /var/lib/cassandra/da= ta/OpsCenter/rollups60/OpsCenter-rollups60-tmp-jb-58656-Data.db > =09at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableW= riter.java:141) > =09at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.= java:164) > =09at org.apache.cassandra.db.compaction.CompactionTask.runWith(Compactio= nTask.java:160) > =09at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwar= eRunnable.java:48) > =09at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java= :28) > =09at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(C= ompactionTask.java:60) > =09at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(A= bstractCompactionTask.java:59) > =09at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(= CompactionManager.java:296) > =09at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java= :28) > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 471) > =09at java.util.concurrent.FutureTask.run(FutureTask.java:262) > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1145) > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:615) > =09at java.lang.Thread.run(Thread.java:724) > Moving back to STC worked to keep the compactions running. > Especialy my own Table i would like to move to LCS. > After a major compaction with STC the move to LCS fails with the same Exc= eption. -- This message was sent by Atlassian JIRA (v6.2#6252)