Return-Path: Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: (qmail 58274 invoked from network); 22 Jul 2010 16:52:43 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 22 Jul 2010 16:52:43 -0000 Received: (qmail 10483 invoked by uid 500); 22 Jul 2010 16:52:43 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 10469 invoked by uid 500); 22 Jul 2010 16:52:43 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 10461 invoked by uid 99); 22 Jul 2010 16:52:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:52:42 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jul 2010 16:52:38 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 0BAED2388A70; Thu, 22 Jul 2010 16:51:45 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r966742 - in /cassandra/trunk: ./ interface/thrift/gen-java/org/apache/cassandra/thrift/ src/java/org/apache/cassandra/db/ src/java/org/apache/cassandra/dht/ src/java/org/apache/cassandra/service/ test/system/ Date: Thu, 22 Jul 2010 16:51:44 -0000 To: commits@cassandra.apache.org From: jbellis@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20100722165145.0BAED2388A70@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: jbellis Date: Thu Jul 22 16:51:44 2010 New Revision: 966742 URL: http://svn.apache.org/viewvc?rev=966742&view=rev Log: merge from 0.6 Modified: cassandra/trunk/ (props changed) cassandra/trunk/CHANGES.txt cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java (props changed) cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java (props changed) cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java cassandra/trunk/src/java/org/apache/cassandra/dht/Range.java cassandra/trunk/src/java/org/apache/cassandra/service/StorageProxy.java cassandra/trunk/test/system/test_thrift_server.py Propchange: cassandra/trunk/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Jul 22 16:51:44 2010 @@ -1,4 +1,4 @@ -/cassandra/branches/cassandra-0.6:922689-964141,965151,965457,965537 +/cassandra/branches/cassandra-0.6:922689-964141,965151,965457,965537,965604 /incubator/cassandra/branches/cassandra-0.3:774578-796573 /incubator/cassandra/branches/cassandra-0.4:810145-834239,834349-834350 /incubator/cassandra/branches/cassandra-0.5:888872-915439 Modified: cassandra/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/cassandra/trunk/CHANGES.txt?rev=966742&r1=966741&r2=966742&view=diff ============================================================================== --- cassandra/trunk/CHANGES.txt (original) +++ cassandra/trunk/CHANGES.txt Thu Jul 22 16:51:44 2010 @@ -63,6 +63,7 @@ dev * avoid aborting commitlog replay prematurely if a flushed-but- not-removed commitlog segment is encountered (CASSANDRA-1297) * failure detection wasn't closing command sockets (CASSANDRA-1221) + * fix duplicate rows being read during mapreduce (CASSANDRA-1142) 0.6.3 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Jul 22 16:51:44 2010 @@ -1,4 +1,4 @@ -/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-964141,965151,965457,965537 +/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Cassandra.java:922689-964141,965151,965457,965537,965604 /incubator/cassandra/branches/cassandra-0.3/interface/gen-java/org/apache/cassandra/service/Cassandra.java:774578-796573 /incubator/cassandra/branches/cassandra-0.4/interface/gen-java/org/apache/cassandra/service/Cassandra.java:810145-834239,834349-834350 /incubator/cassandra/branches/cassandra-0.5/interface/gen-java/org/apache/cassandra/service/Cassandra.java:888872-903502 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Jul 22 16:51:44 2010 @@ -1,4 +1,4 @@ -/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-964141,965151,965457,965537 +/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/Column.java:922689-964141,965151,965457,965537,965604 /incubator/cassandra/branches/cassandra-0.3/interface/gen-java/org/apache/cassandra/service/column_t.java:774578-792198 /incubator/cassandra/branches/cassandra-0.4/interface/gen-java/org/apache/cassandra/service/Column.java:810145-834239,834349-834350 /incubator/cassandra/branches/cassandra-0.5/interface/gen-java/org/apache/cassandra/service/Column.java:888872-903502 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Jul 22 16:51:44 2010 @@ -1,4 +1,4 @@ -/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-964141,965151,965457,965537 +/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/InvalidRequestException.java:922689-964141,965151,965457,965537,965604 /incubator/cassandra/branches/cassandra-0.3/interface/gen-java/org/apache/cassandra/service/InvalidRequestException.java:774578-796573 /incubator/cassandra/branches/cassandra-0.4/interface/gen-java/org/apache/cassandra/service/InvalidRequestException.java:810145-834239,834349-834350 /incubator/cassandra/branches/cassandra-0.5/interface/gen-java/org/apache/cassandra/service/InvalidRequestException.java:888872-903502 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Jul 22 16:51:44 2010 @@ -1,4 +1,4 @@ -/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java:922689-964141,965151,965457,965537 +/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/NotFoundException.java:922689-964141,965151,965457,965537,965604 /incubator/cassandra/branches/cassandra-0.3/interface/gen-java/org/apache/cassandra/service/NotFoundException.java:774578-796573 /incubator/cassandra/branches/cassandra-0.4/interface/gen-java/org/apache/cassandra/service/NotFoundException.java:810145-834239,834349-834350 /incubator/cassandra/branches/cassandra-0.5/interface/gen-java/org/apache/cassandra/service/NotFoundException.java:888872-903502 Propchange: cassandra/trunk/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Thu Jul 22 16:51:44 2010 @@ -1,4 +1,4 @@ -/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java:922689-964141,965151,965457,965537 +/cassandra/branches/cassandra-0.6/interface/thrift/gen-java/org/apache/cassandra/thrift/SuperColumn.java:922689-964141,965151,965457,965537,965604 /incubator/cassandra/branches/cassandra-0.3/interface/gen-java/org/apache/cassandra/service/superColumn_t.java:774578-792198 /incubator/cassandra/branches/cassandra-0.4/interface/gen-java/org/apache/cassandra/service/SuperColumn.java:810145-834239,834349-834350 /incubator/cassandra/branches/cassandra-0.5/interface/gen-java/org/apache/cassandra/service/SuperColumn.java:888872-903502 Modified: cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java?rev=966742&r1=966741&r2=966742&view=diff ============================================================================== --- cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java Thu Jul 22 16:51:44 2010 @@ -975,7 +975,7 @@ public class ColumnFamilyStore implement * @param columnFilter description of the columns we're interested in for each row * @return true if we found all keys we were looking for, otherwise false */ - private boolean getRangeRows(List rows, byte[] superColumn, final AbstractBounds range, int maxResults, IFilter columnFilter) + private void getRangeRows(List rows, byte[] superColumn, final AbstractBounds range, int maxResults, IFilter columnFilter) throws ExecutionException, InterruptedException { final DecoratedKey startWith = new DecoratedKey(range.left, (byte[])null); @@ -1005,17 +1005,20 @@ public class ColumnFamilyStore implement DecoratedKey key = current.key; if (!stopAt.isEmpty() && stopAt.compareTo(key) < 0) - return true; + return; // skip first one if(range instanceof Bounds || !first || !key.equals(startWith)) + { rows.add(current); + if (logger_.isDebugEnabled()) + logger_.debug("scanned " + key); + } first = false; if (rows.size() >= maxResults) - return true; + return; } - return false; } finally { @@ -1043,24 +1046,10 @@ public class ColumnFamilyStore implement throws ExecutionException, InterruptedException { List rows = new ArrayList(); - boolean completed; - if ((range instanceof Bounds || !((Range)range).isWrapAround())) - { - completed = getRangeRows(rows, super_column, range, keyMax, columnFilter); - } - else - { - // wrapped range - Token min = partitioner_.getMinimumToken(); - Range first = new Range(range.left, min); - completed = getRangeRows(rows, super_column, first, keyMax, columnFilter); - if (!completed && min.compareTo(range.right) < 0) - { - Range second = new Range(min, range.right); - getRangeRows(rows, super_column, second, keyMax, columnFilter); - } - } - + assert range instanceof Bounds + || (!((Range)range).isWrapAround() || range.right.equals(StorageService.getPartitioner().getMinimumToken())) + : range; + getRangeRows(rows, super_column, range, keyMax, columnFilter); return rows; } Modified: cassandra/trunk/src/java/org/apache/cassandra/dht/Range.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/dht/Range.java?rev=966742&r1=966741&r2=966742&view=diff ============================================================================== --- cassandra/trunk/src/java/org/apache/cassandra/dht/Range.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/dht/Range.java Thu Jul 22 16:51:44 2010 @@ -131,10 +131,10 @@ public class Range extends AbstractBound */ public Set intersectionWith(Range that) { - if (this.contains(that)) - return rangeSet(that); if (that.contains(this)) return rangeSet(this); + if (this.contains(that)) + return rangeSet(that); boolean thiswraps = isWrapAround(left, right); boolean thatwraps = isWrapAround(that.left, that.right); Modified: cassandra/trunk/src/java/org/apache/cassandra/service/StorageProxy.java URL: http://svn.apache.org/viewvc/cassandra/trunk/src/java/org/apache/cassandra/service/StorageProxy.java?rev=966742&r1=966741&r2=966742&view=diff ============================================================================== --- cassandra/trunk/src/java/org/apache/cassandra/service/StorageProxy.java (original) +++ cassandra/trunk/src/java/org/apache/cassandra/service/StorageProxy.java Thu Jul 22 16:51:44 2010 @@ -697,10 +697,13 @@ public class StorageProxy implements Sto * D, but we don't want any other results from it until after the (D, T] range. Unwrapping so that * the ranges we consider are (D, T], (T, MIN], (MIN, D] fixes this. */ - private static List getRestrictedRanges(AbstractBounds queryRange) + private static List getRestrictedRanges(final AbstractBounds queryRange) { TokenMetadata tokenMetadata = StorageService.instance.getTokenMetadata(); + if (logger.isDebugEnabled()) + logger.debug("computing restricted ranges for query " + queryRange); + List ranges = new ArrayList(); // for each node, compute its intersection with the query range, and add its unwrapped components to our list for (Token nodeToken : tokenMetadata.sortedTokens()) @@ -720,14 +723,23 @@ public class StorageProxy implements Sto // re-sort ranges in ring order, post-unwrapping Comparator comparator = new Comparator() { + // no restricted ranges will overlap so we don't need to worry about inclusive vs exclusive left, + // just sort by raw token position. public int compare(AbstractBounds o1, AbstractBounds o2) { - // no restricted ranges will overlap so we don't need to worry about inclusive vs exclusive left, - // just sort by raw token position. - return o1.left.compareTo(o2.left); + // sort in order that the original query range would see them. + int queryOrder1 = queryRange.left.compareTo(o1.left); + int queryOrder2 = queryRange.left.compareTo(o2.left); + if (queryOrder1 < queryOrder2) + return -1; // o1 comes after query start, o2 wraps to after + if (queryOrder1 > queryOrder2) + return 1; // o2 comes after query start, o1 wraps to after + return o1.left.compareTo(o2.left); // o1 and o2 are on the same side of query start } }; Collections.sort(ranges, comparator); + if (logger.isDebugEnabled()) + logger.debug("Sorted ranges are [" + StringUtils.join(ranges, ", ") + "]"); return ranges; } Modified: cassandra/trunk/test/system/test_thrift_server.py URL: http://svn.apache.org/viewvc/cassandra/trunk/test/system/test_thrift_server.py?rev=966742&r1=966741&r2=966742&view=diff ============================================================================== --- cassandra/trunk/test/system/test_thrift_server.py (original) +++ cassandra/trunk/test/system/test_thrift_server.py Thu Jul 22 16:51:44 2010 @@ -970,6 +970,29 @@ class TestMutations(ThriftTester): assert result[1].columns[0].column.name == 'col1' + def test_wrapped_range_slices(self): + _set_keyspace('Keyspace1') + + def copp_token(key): + # I cheated and generated this from Java + return {'a': '00530000000100000001', + 'b': '00540000000100000001', + 'c': '00550000000100000001', + 'd': '00560000000100000001', + 'e': '00580000000100000001'}[key] + + for key in ['a', 'b', 'c', 'd', 'e']: + for cname in ['col1', 'col2', 'col3', 'col4', 'col5']: + client.insert(key, ColumnParent('Standard1'), Column(cname, 'v-' + cname, Clock(0)), ConsistencyLevel.ONE) + cp = ColumnParent('Standard1') + + result = client.get_range_slices(cp, SlicePredicate(column_names=['col1', 'col3']), KeyRange(start_token=copp_token('e'), end_token=copp_token('e')), ConsistencyLevel.ONE) + assert [row.key for row in result] == ['a', 'b', 'c', 'd', 'e',], [row.key for row in result] + + result = client.get_range_slices(cp, SlicePredicate(column_names=['col1', 'col3']), KeyRange(start_token=copp_token('c'), end_token=copp_token('c')), ConsistencyLevel.ONE) + assert [row.key for row in result] == ['d', 'e', 'a', 'b', 'c',], [row.key for row in result] + + def test_get_slice_by_names(self): _set_keyspace('Keyspace1') _insert_range()