cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bjc (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-781) in a cluster, get_range_slice() does not return all the keys it should
Date Sun, 14 Feb 2010 00:12:28 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833477#action_12833477
] 

bjc commented on CASSANDRA-781:
-------------------------------

Thanks for fixing the patch, but I'm sorry...it still doesn't work right.. :( I found a couple
of weird things. First, sometimes when I ask for 10, it gives more. Second, if I pass start=""
it still doesn't start at the beginning. 

However, this doesn't happen every time I start fresh. Maybe it's depedent on the tokens.
Here are the tokens for the case I show below: 

 INFO 23:38:21,989 Saved Token not found. Using 0njMRYmU9KiXE80d 
 INFO 23:38:20,112 Saved Token not found. Using I8LW8J6h9UCuz0dC 

The first token belongs to the node I am attaching the client to.

This functionality seems surprisingly complicated. Maybe it would help to write down pseudo-code
for the way it should work? I have to admit that I cannot piece it together by reading the
comments in your patch.

Here is a transcript and test: 

First run, looks ok: 

ip-10-212-87-165$ python test_bug_simple3.py 
insert 10b470f49a7c46bd938d784ca4096b63 
insert 47f70aacb3e94e10acaf8e86edac7169 
insert 9a3b7d3b921345bebc4f2bedc1db7c01 
insert c1c9dab59abd4f4ca33ee79f71a179e9 
insert cff81b145faf4648ac8ae001973c6c75 
insert c752d33e5d344312908e5008e6cdae3e 
insert 6e9e32e8b89845bb935d993a9c8bcb13 
insert c286bf2711bc45c1ab033561112c2313 
insert 2dad487ddfa94c81b52c8b4d35d3cb5c 
insert 9c62c7dafdb94dfdbdf52b527bdd2b24 

result 10b470f49a7c46bd938d784ca4096b63 
result 2dad487ddfa94c81b52c8b4d35d3cb5c 
result 47f70aacb3e94e10acaf8e86edac7169 
result 6e9e32e8b89845bb935d993a9c8bcb13 
result 9a3b7d3b921345bebc4f2bedc1db7c01 
result 9c62c7dafdb94dfdbdf52b527bdd2b24 
result c1c9dab59abd4f4ca33ee79f71a179e9 
result c286bf2711bc45c1ab033561112c2313 
result c752d33e5d344312908e5008e6cdae3e 
result cff81b145faf4648ac8ae001973c6c75 
total_keys 10 

Second run, get 18 keys when I asked for 10: 

ip-10-212-87-165$ python test_bug_simple3.py 
insert f873d662dccf46c28080a01286e09ed8 
insert 903776c2f45740389aa52675bf47c7ec 
insert 0e80401a9052405a898d11e5ae874a13 
insert 398d51ba174b4c9db8c25ca6cd2c9454 
insert 50f1cd47dd284ee9b9573b4dfce39134 
insert 20fa43d2365b4dfab9b05a93992315d0 
insert e009d5b76e8840b784fe6b9b649ae1df 
insert 63497f9d63c74b99a681fa2fc52751ac 
insert 824bbcf997de48a99cad174e9e1f1eec 
insert 01c5a6506f4247068660c20338a03bb3 

result 01c5a6506f4247068660c20338a03bb3 
result 0e80401a9052405a898d11e5ae874a13 
result 10b470f49a7c46bd938d784ca4096b63 
result 20fa43d2365b4dfab9b05a93992315d0 
result 2dad487ddfa94c81b52c8b4d35d3cb5c 
result 398d51ba174b4c9db8c25ca6cd2c9454 
result 47f70aacb3e94e10acaf8e86edac7169 
result 50f1cd47dd284ee9b9573b4dfce39134 
result 63497f9d63c74b99a681fa2fc52751ac 
result 6e9e32e8b89845bb935d993a9c8bcb13 
result 824bbcf997de48a99cad174e9e1f1eec 
result 903776c2f45740389aa52675bf47c7ec 
result c1c9dab59abd4f4ca33ee79f71a179e9 
result c286bf2711bc45c1ab033561112c2313 
result c752d33e5d344312908e5008e6cdae3e 
result cff81b145faf4648ac8ae001973c6c75 
result e009d5b76e8840b784fe6b9b649ae1df 
result f873d662dccf46c28080a01286e09ed8 
total_keys 18 

Third run, start at "ca.." even though I pass start="" and all the previous keys remain: 

ip-10-212-87-165$ python test_bug_simple3.py 
insert ca97d7efb63448f8a62d6f7f73044236 
insert 91363a713b714af88ac2191caeea5351 
insert 7b6756d0ab8e450b826b1abc7210d524 
insert e6c6765497af4078b93e1a1470bd3194 
insert e3457f26754c4e7cb7ef606f98e7bb78 
insert 99643eb237ea4ca8b50cac4bb4d58edd 
insert ec3e1f81359b4ae08cfed73899934a93 
insert ae2b990ceb044bf194a879059f823ecf 
insert 2c1494f0ad3d48d2bf4feb33f40cf38e 
insert 0cb1c2e906b64fee89f7729052e0810e 

result ae2b990ceb044bf194a879059f823ecf 
result c1c9dab59abd4f4ca33ee79f71a179e9 
result c286bf2711bc45c1ab033561112c2313 
result c752d33e5d344312908e5008e6cdae3e 
result ca97d7efb63448f8a62d6f7f73044236 
result cff81b145faf4648ac8ae001973c6c75 
result e009d5b76e8840b784fe6b9b649ae1df 
result e3457f26754c4e7cb7ef606f98e7bb78 
result e6c6765497af4078b93e1a1470bd3194 
result ec3e1f81359b4ae08cfed73899934a93 
total_keys 10 
ip-10-212-87-165$ 

Here is another run (different tokens):

 INFO 00:04:17,105 Saved Token not found. Using Iw1khrAgM5sd6WnX
 INFO 00:04:15,795 Saved Token not found. Using IgLbq912n2xEP99G

In this case I don't see the problem where I get back more keys than I asked for, but I don't
get the 10 lowest keys in the second request. 10 are returned, but they are not ordered consistently
with what I know is in the db.

First run, notice key "893.." is inserted:

ip-10-212-87-165$ python test_bug_simple3.py 
insert 7be5d87bc45843cfaffd36fd654aee53
insert 8ef4727d83474570aa2111bee3929a5f
insert 9a9a91b6b662430092db0209d63a5c9e
insert e45afe1f0e364012acd0dead5b75ea13
insert 10171c87634842aea4f16d46d611c435
insert 10b6f92ac6a447088a82c4ec13056f1e
insert c1acbde9ae454ea2819322975322206b
insert 89352cf117dd4cb9ab935cbb5f230ba0
insert 0b5d924f04174459969594d6293b9aca
insert e2471db4d8f445f2b0c36f3b2a5bb650

result 0b5d924f04174459969594d6293b9aca
result 10171c87634842aea4f16d46d611c435
result 10b6f92ac6a447088a82c4ec13056f1e
result 7be5d87bc45843cfaffd36fd654aee53
result 89352cf117dd4cb9ab935cbb5f230ba0
result 8ef4727d83474570aa2111bee3929a5f
result 9a9a91b6b662430092db0209d63a5c9e
result c1acbde9ae454ea2819322975322206b
result e2471db4d8f445f2b0c36f3b2a5bb650
result e45afe1f0e364012acd0dead5b75ea13
total_keys 10

Second run, notice the results start with "0.." but "893.." is not returned (though "b2.."
is, and other higher keys):

ip-10-212-87-165$ python test_bug_simple3.py 
insert 85d282dfa03a466eb51d03f4eb5dacd5
insert a61e7757eaed4ef79fc7bf35f47843f7
insert 9b9b6e3f22994827b0dddcc16105ff7d
insert 880b1644636845d8b1c92faf1f6d8484
insert 5d1d7d7b26ec4540a89c027bccc17e06
insert 4df05d38950f44b29df604c165e1148f
insert 0c9f818aa28a47fb832b6a0929b94280
insert 7e7b829c136046a88b120a4a373d9a6b
insert b2d497713f9342de85bb31b5c0e69af6
insert 80e250253f1942aaab2e9b49880918c0

result 0b5d924f04174459969594d6293b9aca
result 0c9f818aa28a47fb832b6a0929b94280
result 10171c87634842aea4f16d46d611c435
result 10b6f92ac6a447088a82c4ec13056f1e
result 4df05d38950f44b29df604c165e1148f
result a61e7757eaed4ef79fc7bf35f47843f7
result b2d497713f9342de85bb31b5c0e69af6
result c1acbde9ae454ea2819322975322206b
result e2471db4d8f445f2b0c36f3b2a5bb650
result e45afe1f0e364012acd0dead5b75ea13
total_keys 10
ip-10-212-87-165$ 

I found another issue with "nodetool ring" which might be related to the patch: 

$ sudo bin/nodetool -h localhost ring 
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException 
        at $Proxy0.getRangeToEndPointMap(Unknown Source) 
        at org.apache.cassandra.tools.NodeProbe.getRangeToEndPointMap(NodeProbe.java:151)

        at org.apache.cassandra.tools.NodeCmd.printRing(NodeCmd.java:74) 
        at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:403) 
Caused by: java.rmi.UnmarshalException: error unmarshalling return; nested exception is: 
        java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException:
org.apache.cassandra.dht.OrderPreservingPartitioner 
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173) 
        at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) 
        at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) 
        at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993)

        at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)

        ... 4 more 
Caused by: java.io.WriteAbortedException: writing aborted; java.io.NotSerializableException:
org.apache.cassandra.dht.OrderPreservingPartitioner 
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1333) 
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947) 
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871) 
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) 
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) 
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) 
        at java.util.HashMap.readObject(HashMap.java:1029) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597) 
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974) 
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849) 
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) 
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) 
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) 
        at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306) 
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:155) 
        ... 8 more 
Caused by: java.io.NotSerializableException: org.apache.cassandra.dht.OrderPreservingPartitioner

        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1156) 
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509) 
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474) 
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392) 
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) 
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326) 
        at java.util.HashMap.writeObject(HashMap.java:1000) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597) 
        at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945) 
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461) 
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392) 
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150) 
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326) 
        at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274) 
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315) 
        at sun.rmi.transport.Transport$1.run(Transport.java:159) 
        at java.security.AccessController.doPrivileged(Native Method) 
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155) 
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) 
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)

        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:619) 
$ 


Here is the test I ran above: 

import uuid 

from thrift import Thrift 
from thrift.transport import TTransport 
from thrift.transport import TSocket 
from thrift.protocol.TBinaryProtocol import TBinaryProtocolAccelerated 

import sys 
sys.path.insert(0,'/usr/local/cassandra/interface/thrift/gen-py') 

from cassandra import Cassandra 
from cassandra.ttypes import * 

socket = TSocket.TSocket("10.212.87.165", 9160) 
transport = TTransport.TBufferedTransport(socket) 
protocol = TBinaryProtocol.TBinaryProtocolAccelerated(transport) 
client = Cassandra.Client(protocol) 

transport.open() 

ks = "Keyspace1" 
cf = "Super1" 
path = ColumnPath(cf, "foo", "is") 
value = "cool" 

for i in xrange(10): 
    key = uuid.uuid4().hex 
    print "insert", key 
    client.insert(ks, key, path, value, 0, ConsistencyLevel.ONE) 

print 

parent = ColumnParent(column_family=cf) 
slice_range = SliceRange(start="key", finish="key") 
predicate = SlicePredicate(slice_range=slice_range) 

total_keys = 0 

result = client.get_range_slice(ks, parent, predicate, "", "", 10, ConsistencyLevel.ONE) 
for row in result: 
    total_keys += 1 
    print "result", row.key 

print "total_keys", total_keys 




> in a cluster, get_range_slice() does not return all the keys it should
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-781
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-781
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 0.5
>         Environment: Debian 5 lenny on EC2, Gentoo linux, Windows XP
>            Reporter: bjc
>            Assignee: Jonathan Ellis
>             Fix For: 0.5, 0.6
>
>         Attachments: 781.txt
>
>
> get_range_slice() does not return the same set of keys as get_key_range() in 0.5.0 final.
> I posted a program to reproduce the behavior:
> http://www.mail-archive.com/cassandra-dev@incubator.apache.org/msg01474.html
> Apparently, you must have more than one node to get the behavior. Also, it may depend
on the locations of the nodes on the ring.. I.e., if you don't generate enough keys randomly,
then by chance they could all fall on the same host and you might not see the behavior, although
I was able to get it to happen using only 2 nodes and 10 keys.
> Here are the other emails describing the issue:
> http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg02423.html

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message