Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AB9106E6F for ; Wed, 18 May 2011 12:41:59 +0000 (UTC) Received: (qmail 9352 invoked by uid 500); 18 May 2011 12:41:57 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 9316 invoked by uid 500); 18 May 2011 12:41:57 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 9304 invoked by uid 99); 18 May 2011 12:41:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 12:41:57 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Ignace.Desimpel@nuance.com designates 198.71.73.25 as permitted sender) Received: from [198.71.73.25] (HELO mx2.nuance.com) (198.71.73.25) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 May 2011 12:41:50 +0000 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgoBADe+000KAQS+/2dsb2JhbACXTI9DiHC/DIYZBJRJFIor X-IronPort-AV: E=Sophos;i="4.65,231,1304308800"; d="scan'208";a="236713032" Received: from unknown (HELO bn-exchbh1.nuance.com) ([10.1.4.190]) by mx2.nuance.com with ESMTP; 18 May 2011 08:41:00 -0400 Received: from eu-exchbh2.nuance.com ([10.164.0.126]) by bn-exchbh1.nuance.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 18 May 2011 08:41:00 -0400 Received: from EU-EXCH02.nuance.com ([10.164.0.154]) by eu-exchbh2.nuance.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 18 May 2011 14:40:57 +0200 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: AssertionError Date: Wed, 18 May 2011 14:40:56 +0200 Message-ID: <15FEFA0410B1AA4F9775D72F0279A368C1AE6A@EU-EXCH02.nuance.com> In-Reply-To: <15FEFA0410B1AA4F9775D72F0279A368C1ABCF@EU-EXCH02.nuance.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: AssertionError Thread-Index: AcwUkSZY4jBzrBUAQCaWDXzbB9eNfgABIuVAADA7xbA= References: <15FEFA0410B1AA4F9775D72F0279A368C1A941@EU-EXCH02.nuance.com><3A0D4D0E-843A-4DC9-A6F1-1884CF12E458@thelastpickle.com><15FEFA0410B1AA4F9775D72F0279A368C1AB1B@EU-EXCH02.nuance.com><15FEFA0410B1AA4F9775D72F0279A368C1AB4B@EU-EXCH02.nuance.com> <15FEFA0410B1AA4F9775D72F0279A368C1ABCF@EU-EXCH02.nuance.com> From: "Desimpel, Ignace" To: X-OriginalArrivalTime: 18 May 2011 12:40:57.0893 (UTC) FILETIME=[D6061550:01CC1558] X-Virus-Checked: Checked by ClamAV on apache.org Hi Sylvain, I did the upgrade from 0.7.4 to 0.7.5 and the exception does not occur = anymore (on Windows ...). Thanks for pointing me to the bug fix. >From the 0.7.5 version I upgraded to the 0.7.6 version, and this is also = OK, without any code changes and by still keeping the same data files = generate with the 0.7.4 version. Could you still give me a comment on the question regarding the = AbstractType class change? To be on the save side, I could simply make = new array backed byte buffers (that is what I need). But I ask the = question because I want to avoid allocating any object if it is not = really needed since I know that I will query for a lot of data of that = type. Ignace -----Original Message----- From: Desimpel, Ignace [mailto:Ignace.Desimpel@nuance.com]=20 Sent: Tuesday, May 17, 2011 3:33 PM To: user@cassandra.apache.org Subject: RE: AssertionError Seems like the AbstractType class has changed going from 0.7.4 to 0.7.5. = It is now required to implement a compose and decompose method. Already = did that, and it starts up with the 0.7.5 code using the 0.7.4 data and = configuration (using a smaller extra test database) Below I made a = sample implementations to illustrate another question : On the compose = method , can I simply create my own AbstractType class and use the given = ByteBuffer. Or like in the decompose example, do I need to duplicate the = ByteBuffer or could the paramT object be reused or should I make a = complete copy? @Override public Object compose(ByteBuffer paramByteBuffer){ ReverseCFFloatValues oNew =3D new ReverseCFFloatValues(); oNew.paramByteBuffer =3D paramByteBuffer; return oNew; } @Override public ByteBuffer decompose(Object paramT){ return ((ReverseCFFloatValues)paramT).paramByteBuffer.duplicate(); } -----Original Message----- From: Sylvain Lebresne [mailto:sylvain@datastax.com] Sent: Tuesday, May 17, 2011 2:50 PM To: user@cassandra.apache.org Subject: Re: AssertionError On Tue, May 17, 2011 at 1:46 PM, Desimpel, Ignace = wrote: > Ok, I will do that (next test will be done on some linux boxes being = installed now, but at this time I need to gone with the current windows = setup). > Question : Can I use the 0.7.4 data files as is? Do I need to backup = the datafiles in order to be able to get back to the 0.7.4 version if = needed? Yes you can use 0.7.4 data files as is. And I can't think of a reason = why you should have problem getting back to 0.7.4 if needed, though = snapshotting before cannot hurt. > > Ignace > > -----Original Message----- > From: Sylvain Lebresne [mailto:sylvain@datastax.com] > Sent: Tuesday, May 17, 2011 1:16 PM > To: user@cassandra.apache.org > Subject: Re: AssertionError > > First thing to do would be to update to 0.7.5. > > The assertionError you're running into is a assertion where we check = if a skipBytes did skip all the bytes we had ask him to. As it turns = out, the spec for skipBytes authorize it to not skip all the bytes asked = even with no good reason. I'm pretty sure on a linux box skipBytes on a = file will always read the number of asked bytes unless it reaches EOF, = but I see you're running windows, so who knows what can happen. > > Anyway, long story short, it's a "bug" in 0.7.4 that has been fixed in = 0.7.5. If you still run into this in 0.7.5 at least we'll know it's = something else (and we will have a more helpful error message). > > -- > Sylvain > > On Tue, May 17, 2011 at 12:41 PM, Desimpel, Ignace = wrote: >> I use a custom comparator class. So I think there is a high chance=20 >> that I do something wrong there. I was thinking that the stack trace=20 >> could give a clue and help me on the way, maybe because some already = got the same error. >> >> >> >> Anyway, here is some more information you requested. >> >> >> >> Yaml definition : >> >> name: ForwardStringValues >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 column_type: Super >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 compare_with: >> be.landc.services.search.server.db.cassandra.node.ForwardCFStringValu >> e >> s >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 compare_subcolumns_with: BytesType >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 keys_cached: 100000 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 rows_cached: 0 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 comment: Stores the values of functions = returning string >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 memtable_throughput_in_mb: 64 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 memtable_operations_in_millions: 15 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 min_compaction_threshold: 2 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0 max_compaction_threshold: 5 >> >> >> >> Column Family: ForwardStringValues >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 SSTable count: 8 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Space used (live): = 131311776690 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Space used (total): = 131311776690 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Columns Count: = 0 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Data Size: 0 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Memtable Switch Count: = 0 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Read Count: 1 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Read Latency: 404.890 = ms. >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Write Count: 0 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Write Latency: NaN ms. >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Pending Tasks: 0 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache capacity: = 100000 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache size: 8 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Key cache hit rate: 1.0 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Row cache: disabled >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Compacted row minimum = size: 150 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Compacted row maximum = size: 7152383774 >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Compacted row mean = size: 3064535 >> >> >> >> No secondary indexes. >> >> Total database disk size 823 Gb >> >> disk_access_mode: auto on 64 bit windows os >> >> partitioner: org.apache.cassandra.dht.ByteOrderedPartitioner >> >> Data was stored over a period of 5 days. >> >> Cassandra 0.7.4 was running as an embedded server. >> >> Batch insert, using the StorageProxy.mutate. >> >> No errors were logged during the batch insert period. >> >> The row key is a string representation of a positive integer value. >> >> The same row key is used during many different mutate calls, but all=20 >> super column names are different for each call. >> >> The column name of the super class stored =A0is composed of the 32=20 >> bytes and the bytes of 2 integer (positive and negative) values and=20 >> the bytes (UTF8) of the string value :[32 bytes][4 int bytes][4 int=20 >> bytes][string bytes] >> >> The custom comparator class ...ForwardCFStringValues sorts the names=20 >> by first sorting the string , then the 32 bytes, and then the two=20 >> integer values >> >> For each column name two subcolumns are inserted with fixed name and=20 >> some small binary value (about 40 bytes) >> >> >> >> The query : >> >> Get_slice using thrift. >> >> Params : >> >> =A0=A0Row key : the string representation of the positive integer = String '1788' >> thus hex values 31 37 38 38 >> >> =A0 ColumnParent : the column family ForwardStringValues >> >> =A0 SlicePredicate : SlicePredicate(slice_range:SliceRange(start:00 = 00 >> 00 00 >> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >> 00 00 >> 00 00 00 FF FF FF FF FF FF FF FF 55 52 49 4E 41 52 59 20 54 52 41 43 >> 54 20 >> 49 4E 46 45 43 54 49 4F 4E, finish:7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F=20 >> 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F FF FF=20 >> FF 7F FF FF FF 7F 55 52 49 4E 41 52 59 20 54 52 41 43 54 20 49 4E 46 >> 45 >> 43 54 49 4F 4E, reversed:false, count:10000)) >> >> >> >> This SlicePredicate is supposed to fetch all the columns with the=20 >> string '55 >> 52 49 4E 41 52 59 20 54 52 41 43 54 20 49 4E 46 45 43 54 49 4F 4E' >> regardless of the other bytes in the column name. So the start and=20 >> finish have the same string bytes. The rest of the bytes for the=20 >> start values are set to the lowest possible value (32 zero bytes and=20 >> the bytes FFFFFFFF representing the integer value -1) , the finish is = >> set the highest possible value (32 bytes with value 7F, ...) >> >> >> >> I tested the same code but with a small data set and all seemed to be = OK. >> Even on the same database I get back results without exception if I=20 >> use different String values. I'm almost sure that there should be=20 >> columns with that string. If the string is not present I don't get = the error. >> >> >> >> >> >> >> >> >> >> From: Aaron Morton [mailto:aaron@thelastpickle.com] >> Sent: Monday, May 16, 2011 11:33 PM >> To: user@cassandra.apache.org >> Subject: Re: AssertionError >> >> >> >> The code is trying to follow the column index for a row in an=20 >> sstable, but it cannot skip as many bytes as it would like to to get = to the column. >> Helpfully the help says running out of bytes is only one of the=20 >> reasons why this could happen:) >> >> >> >> Can you provide some more information about the query and the data,=20 >> and also the upgrade history for your cluster. >> >> >> >> Thanks >> >> Aaron >> >> On 17/05/2011, at 3:07 AM, "Desimpel, Ignace" >> >> wrote: >> >> Environment : java 64 bit server, java client, thrift get_slice=20 >> method, Cassandra 0.7.4, single node >> >> Depending on the data I pass for a query on a CF I get the following=20 >> listed below. Any suggestions what could be wrong based on the stack = trace? >> >> >> >> java.lang.AssertionError >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBloc >> k >> Fetcher.getNextBlock(IndexedSliceReader.java:176) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext >> ( >> IndexedSliceReader.java:120) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext >> ( >> IndexedSliceReader.java:48) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractI >> t >> erator.java:136) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.j >> a >> va:131) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(S >> S >> TableSliceIterator.java:108) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.commons.collections.iterators.CollatingIterator.set(Collat >> i >> ngIterator.java:282) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.commons.collections.iterators.CollatingIterator.least(Coll >> a >> tingIterator.java:325) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.commons.collections.iterators.CollatingIterator.next(Colla >> t >> ingIterator.java:229) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingItera >> t >> or.java:68) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractI >> t >> erator.java:136) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.j >> a >> va:131) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns >> ( >> SliceQueryFilter.java:116) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(Que >> r >> yFilter.java:130) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa >> m >> ilyStore.java:1368) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil >> y >> Store.java:1245) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil >> y >> Store.java:1173) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.Table.getRow(Table.java:333) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadComm >> a >> nd.java:63) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThr >> o >> w(StorageProxy.java:453) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:3 >> 0 >> ) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec >> u >> tor.java:886) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at >> = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. >> java:908) >> >> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 at = java.lang.Thread.run(Thread.java:662) >> >> >> >> Ignace Desimpel >