Return-Path: Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: (qmail 91055 invoked from network); 5 Apr 2011 14:26:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Apr 2011 14:26:47 -0000 Received: (qmail 54141 invoked by uid 500); 5 Apr 2011 14:26:47 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 54091 invoked by uid 500); 5 Apr 2011 14:26:47 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 54083 invoked by uid 99); 5 Apr 2011 14:26:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Apr 2011 14:26:47 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Apr 2011 14:26:43 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 55CDF935B9 for ; Tue, 5 Apr 2011 14:26:06 +0000 (UTC) Date: Tue, 5 Apr 2011 14:26:06 +0000 (UTC) From: "Tey Kar Shiang (JIRA)" To: commits@cassandra.apache.org Message-ID: <2019907899.34985.1302013566347.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <1008219186.18368.1301383025784.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Issue Comment Edited] (CASSANDRA-2401) getColumnFamily() return null, which is not checked in ColumnFamilyStore.java scan() method, causing Timeout Exception in query MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015932#comment-13015932 ] Tey Kar Shiang edited comment on CASSANDRA-2401 at 4/5/11 2:24 PM: ------------------------------------------------------------------- hi, yes. it seems to me so. Here, we create a table "FileMap", in which we store columns e.g. "content", "authorID", "Version", "Modified Time", "File Type", etc. Among them, sorted indices are "authorID" (as UserIndex), "File Type", "Modified Time", and "CassType"; where CassType means generally 'file type' here in our case. It is not used though. 
03/30/2011  09:34 AM        11,366,878 FileMap-f-53-Data.db
03/30/2011  09:34 AM            78,496 FileMap-f-53-Filter.db
03/30/2011  09:34 AM           735,930 FileMap-f-53-Index.db
03/30/2011  09:34 AM             4,264 FileMap-f-53-Statistics.db
03/30/2011  05:37 PM             4,055 FileMap-f-54-Data.db
03/30/2011  05:37 PM                40 FileMap-f-54-Filter.db
03/30/2011  05:37 PM               270 FileMap-f-54-Index.db
03/30/2011  05:37 PM             4,264 FileMap-f-54-Statistics.db
04/04/2011  04:07 PM            24,068 FileMap-f-55-Data.db
04/04/2011  04:07 PM               200 FileMap-f-55-Filter.db
04/04/2011  04:07 PM             1,746 FileMap-f-55-Index.db
04/04/2011  04:07 PM             4,264 FileMap-f-55-Statistics.db
04/04/2011  04:07 PM           961,808 FileMap.CassTypeIndex-f-53-Data.db
04/04/2011  04:07 PM             1,936 FileMap.CassTypeIndex-f-53-Filter.db
04/04/2011  04:07 PM                11 FileMap.CassTypeIndex-f-53-Index.db
04/04/2011  04:07 PM             4,264 FileMap.CassTypeIndex-f-53-Statistics.db
03/29/2011  02:52 PM           961,386 FileMap.FileTypeIndex-f-50-Data.db
03/29/2011  02:52 PM             1,936 FileMap.FileTypeIndex-f-50-Filter.db
03/29/2011  02:52 PM                11 FileMap.FileTypeIndex-f-50-Index.db
03/29/2011  02:52 PM             4,264 FileMap.FileTypeIndex-f-50-Statistics.db
03/30/2011  05:37 PM               404 FileMap.FileTypeIndex-f-51-Data.db
03/30/2011  05:37 PM                16 FileMap.FileTypeIndex-f-51-Filter.db
03/30/2011  05:37 PM                11 FileMap.FileTypeIndex-f-51-Index.db
03/30/2011  05:37 PM             4,264 FileMap.FileTypeIndex-f-51-Statistics.db
04/04/2011  04:07 PM             2,358 FileMap.FileTypeIndex-f-52-Data.db
04/04/2011  04:07 PM                16 FileMap.FileTypeIndex-f-52-Filter.db
04/04/2011  04:07 PM                11 FileMap.FileTypeIndex-f-52-Index.db
04/04/2011  04:07 PM             4,264 FileMap.FileTypeIndex-f-52-Statistics.db
03/29/2011  02:52 PM         3,298,947 FileMap.ModifiedIndex-f-50-Data.db
03/29/2011  02:52 PM            78,016 FileMap.ModifiedIndex-f-50-Filter.db
03/29/2011  02:52 PM           731,106 FileMap.ModifiedIndex-f-50-Index.db
03/29/2011  02:52 PM             4,264 FileMap.ModifiedIndex-f-50-Statistics.db
03/30/2011  05:37 PM             2,065 FileMap.ModifiedIndex-f-51-Data.db
03/30/2011  05:37 PM                64 FileMap.ModifiedIndex-f-51-Filter.db
03/30/2011  05:37 PM               450 FileMap.ModifiedIndex-f-51-Index.db
03/30/2011  05:37 PM             4,264 FileMap.ModifiedIndex-f-51-Statistics.db
04/04/2011  04:07 PM            13,835 FileMap.ModifiedIndex-f-52-Data.db
04/04/2011  04:07 PM               328 FileMap.ModifiedIndex-f-52-Filter.db
04/04/2011  04:07 PM             3,006 FileMap.ModifiedIndex-f-52-Index.db
04/04/2011  04:07 PM             4,264 FileMap.ModifiedIndex-f-52-Statistics.db
04/04/2011  04:07 PM           962,874 FileMap.UserIndex-f-53-Data.db
04/04/2011  04:07 PM             1,936 FileMap.UserIndex-f-53-Filter.db
04/04/2011  04:07 PM               420 FileMap.UserIndex-f-53-Index.db
04/04/2011  04:07 PM             4,264 FileMap.UserIndex-f-53-Statistics.db

In the search, we are using IndexClause as:
		ByteBuffer field_author = ByteBuffer.wrap(new byte[]{'a'});
		ByteBuffer author_1 = IntegerSerializer.get().toByteBuffer(1);
		
		ByteBuffer file_type = ByteBuffer.wrap(new byte[]{'t'});
		ByteBuffer filetype_3 = ByteBuffer.wrap(new byte[]{3}); //file type 3
		
		IndexClause indexClause = new IndexClause();
		indexClause.setCount(3000);
		ArrayList expressions = new ArrayList();
		expressions.add(new IndexExpression(field_author, IndexOperator.EQ, author_1)); //user ID = 1
		expressions.add(new IndexExpression(file_type, IndexOperator.EQ, filetype_3)); //file type = 3
		
		indexClause.setExpressions(expressions);
		indexClause.setStart_key(new byte[]{});
In the search, it scans all the indices from "FileMap.UserIndex", within which there seems having a key (index) which is not found in the table "FileMap"; and I roughly get that it breaks at data retrieval with "FileMap-f-53-Data", when the position for the key is not found / available in "FileMap-f-53-Data". was (Author: karshiang): hi, yes. it seems to me so. Here, we create a table "FileMap", in which we store columns e.g. "content", "authorID", "Version", "Modified Time", "File Type", etc. Among them, sorted indices are "authorID" (as UserIndex), "File Type", "Modified Time", and "CassType"; where CassType means generally 'file type' here in our case. It is not used though. 03/30/2011 09:34 AM 11,366,878 FileMap-f-53-Data.db 03/30/2011 09:34 AM 78,496 FileMap-f-53-Filter.db 03/30/2011 09:34 AM 735,930 FileMap-f-53-Index.db 03/30/2011 09:34 AM 4,264 FileMap-f-53-Statistics.db 03/30/2011 05:37 PM 4,055 FileMap-f-54-Data.db 03/30/2011 05:37 PM 40 FileMap-f-54-Filter.db 03/30/2011 05:37 PM 270 FileMap-f-54-Index.db 03/30/2011 05:37 PM 4,264 FileMap-f-54-Statistics.db 04/04/2011 04:07 PM 24,068 FileMap-f-55-Data.db 04/04/2011 04:07 PM 200 FileMap-f-55-Filter.db 04/04/2011 04:07 PM 1,746 FileMap-f-55-Index.db 04/04/2011 04:07 PM 4,264 FileMap-f-55-Statistics.db 04/04/2011 04:07 PM 961,808 FileMap.CassTypeIndex-f-53-Data.db 04/04/2011 04:07 PM 1,936 FileMap.CassTypeIndex-f-53-Filter.db 04/04/2011 04:07 PM 11 FileMap.CassTypeIndex-f-53-Index.db 04/04/2011 04:07 PM 4,264 FileMap.CassTypeIndex-f-53-Statistics.db 03/29/2011 02:52 PM 961,386 FileMap.FileTypeIndex-f-50-Data.db 03/29/2011 02:52 PM 1,936 FileMap.FileTypeIndex-f-50-Filter.db 03/29/2011 02:52 PM 11 FileMap.FileTypeIndex-f-50-Index.db 03/29/2011 02:52 PM 4,264 FileMap.FileTypeIndex-f-50-Statistics.db 03/30/2011 05:37 PM 404 FileMap.FileTypeIndex-f-51-Data.db 03/30/2011 05:37 PM 16 FileMap.FileTypeIndex-f-51-Filter.db 03/30/2011 05:37 PM 11 FileMap.FileTypeIndex-f-51-Index.db 03/30/2011 05:37 PM 4,264 FileMap.FileTypeIndex-f-51-Statistics.db 04/04/2011 04:07 PM 2,358 FileMap.FileTypeIndex-f-52-Data.db 04/04/2011 04:07 PM 16 FileMap.FileTypeIndex-f-52-Filter.db 04/04/2011 04:07 PM 11 FileMap.FileTypeIndex-f-52-Index.db 04/04/2011 04:07 PM 4,264 FileMap.FileTypeIndex-f-52-Statistics.db 03/29/2011 02:52 PM 3,298,947 FileMap.ModifiedIndex-f-50-Data.db 03/29/2011 02:52 PM 78,016 FileMap.ModifiedIndex-f-50-Filter.db 03/29/2011 02:52 PM 731,106 FileMap.ModifiedIndex-f-50-Index.db 03/29/2011 02:52 PM 4,264 FileMap.ModifiedIndex-f-50-Statistics.db 03/30/2011 05:37 PM 2,065 FileMap.ModifiedIndex-f-51-Data.db 03/30/2011 05:37 PM 64 FileMap.ModifiedIndex-f-51-Filter.db 03/30/2011 05:37 PM 450 FileMap.ModifiedIndex-f-51-Index.db 03/30/2011 05:37 PM 4,264 FileMap.ModifiedIndex-f-51-Statistics.db 04/04/2011 04:07 PM 13,835 FileMap.ModifiedIndex-f-52-Data.db 04/04/2011 04:07 PM 328 FileMap.ModifiedIndex-f-52-Filter.db 04/04/2011 04:07 PM 3,006 FileMap.ModifiedIndex-f-52-Index.db 04/04/2011 04:07 PM 4,264 FileMap.ModifiedIndex-f-52-Statistics.db 04/04/2011 04:07 PM 962,874 FileMap.UserIndex-f-53-Data.db 04/04/2011 04:07 PM 1,936 FileMap.UserIndex-f-53-Filter.db 04/04/2011 04:07 PM 420 FileMap.UserIndex-f-53-Index.db 04/04/2011 04:07 PM 4,264 FileMap.UserIndex-f-53-Statistics.db In the search, we are using IndexClause as: ByteBuffer field_author = ByteBuffer.wrap(new byte[]{'a'}); ByteBuffer author_1 = IntegerSerializer.get().toByteBuffer(1); ByteBuffer file_type = ByteBuffer.wrap(new byte[]{'t'}); ByteBuffer filetype_3 = ByteBuffer.wrap(new byte[]{3}); //file type 3 IndexClause indexClause = new IndexClause(); indexClause.setCount(3000); ArrayList expressions = new ArrayList(); expressions.add(new IndexExpression(field_author, IndexOperator.EQ, author_1)); //user ID = 1 expressions.add(new IndexExpression(file_type, IndexOperator.EQ, filetype_3)); //file type = 3 indexClause.setExpressions(expressions); indexClause.setStart_key(new byte[]{}); In the search, it scans all the indices from "FileMap.UserIndex", within which there seems having a key (index) which is not found in the table "FileMap"; and I roughly get that it breaks at data retrieval with "FileMap-f-53-Data", when the position for the key is not found / available in "FileMap-f-53-Data". > getColumnFamily() return null, which is not checked in ColumnFamilyStore.java scan() method, causing Timeout Exception in query > ------------------------------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-2401 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2401 > Project: Cassandra > Issue Type: Bug > Affects Versions: 0.7.4 > Environment: Hector 0.7.0-28, Cassandra 0.7.4, Windows 7, Eclipse > Reporter: Tey Kar Shiang > > ColumnFamilyStore.java, line near 1680, "ColumnFamily data = getColumnFamily(new QueryFilter(dk, path, firstFilter))", the data is returned null, causing NULL exception in "satisfies(data, clause, primary)" which is not captured. The callback got timeout and return a Timeout exception to Hector. > The data is empty, as I traced, I have the the columns Count as 0 in removeDeletedCF(), which return the null there. (I am new and trying to understand the logics around still). Instead of crash to NULL, could we bypass the data? > About my test: > A stress-test program to add, modify and delete data to keyspace. I have 30 threads simulate concurrent users to perform the actions above, and do a query to all rows periodically. I have Column Family with rows (as File) and columns as index (e.g. userID, fileType). > No issue on the first day of test, and stopped for 3 days. I restart the test on 4th day, 1 of the users failed to query the files (timeout exception received). Most of the users are still okay with the query. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira