incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <Dean.Hil...@nrel.gov>
Subject Re: indexing question related to playOrm on github
Date Fri, 17 Aug 2012 13:30:15 GMT
I am not sure what you mean by play with the timestamp.  I think this works without playing
with the timestamp(thanks for you help as it got me here).

 1.  On a scan I hit <mike><pk>
 2.  I end up looking up the pk
 3.  I compare the value in the row with the indexed value "mike" but I see the row with that
pk has Sam not Mike
 4.  I now know I can discard this result as a false positive.  I also know my index has duplicates.
 5.  I kick off a job to scan the complete index now AND read in each pk row of the index
comparing indexed value with the actual value in the row to fix the index.

I think that might work pretty well.

Thanks,
Dean

From: aaron morton <aaron@thelastpickle.com<mailto:aaron@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Thursday, August 16, 2012 4:55 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: indexing question related to playOrm on github

 I am not sure synchronization fixes thatŠŠIt would be kind of
nice if the column <bill><65> would not actually be removed until after
all servers are eventually consistent...
Not sure thats possible.

You can either serialise updating your custom secondary index on the client site or resolve
the inconsistency on read.

Not sure this fits with your workload but as an e.g. when you read from the index, if you
detect multiple row PK's resolve the issue on the client and leave the data in cassandra as
is. Then queue a job that will read the row and try to repair it's index entries. When repairing
the index entry play with the timestamp so any deletions you make only apply to the column
as it was when you saw the error.

Hope that helps.


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 17/08/2012, at 12:47 AM, "Hiller, Dean" <Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov>>
wrote:

Maybe this would be a special type of column family that could contain
these as my other tables definitely don't want the feature below by the
way.

Dean

On 8/16/12 6:29 AM, "Hiller, Dean" <Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov>>
wrote:

Yes, the synch may work, and no, I do "not" want a transactionŠI want a
different kind of eventually consistent

That might work.
Let's say server 1 sends a mutation (65 is the pk)
Remove: <bill><65>  Add <tim><65>
Server 2 also sends a mutation (65 is the pk)
Remove: <bill><65> Add <mike><65>

What everyone does not want is to end up with a row that has <tim><65>
and <mike><65>.  With the wide row pattern, we would like to have ONE or
the other.  I am not sure synchronization fixes thatŠŠIt would be kind of
nice if the column <bill><65> would not actually be removed until after
all servers are eventually consistent AND would keep a reference to the
add that was happening so that when it goes to resolve eventually
consistent between the servers, it would see that <mike><65> is newer and
it would decide to drop the first add completely.

Ie. In a full process it might look like this
Cassandra node 1 receives remove <bill><65>, add <tim><65> AND in
the
remove column stores info about the add <tim><65> until eventual
consistency is completed
Cassandra node 2 one ms later receives remove <bill><65> and <tim><65>
AND in the remove column stores info about the add <tim><65> until
eventual consistency is completed
Eventual consistency starts comparing node 1 and node 2 and finds
<bill><65> is being removed by different servers and finds add info
attached to that.  ONLY THE LAST add info is acknowledged and it makes
the row consistent across the cluster.

That makes everyone's wide row indexing pattern tend to get less corrupt
over time.

Thanks,
Dean


From: aaron morton
<aaron@thelastpickle.com<mailto:aaron@thelastpickle.com><mailto:aaron@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>>
Date: Wednesday, August 15, 2012 8:26 PM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>"
<user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org>>
Subject: Re: indexing question related to playOrm on github

1.  Can playOrm be listed on cassandra's list of ORMs?  It supports a
JQL/HQL query on a trillion rows in under 100ms (partitioning is the
trick so you can JQL a partition)
No sure if we have an ORM specific page. If it's a client then feel free
to add it to http://wiki.apache.org/cassandra/ClientOptions

I was wondering if cassandra has or will ever support eventual constancy
where it keeps both the REMOVE AND the ADD together such until it is on
all 3 replicated nodes and in resolving the consistency would end up with
an index that only has the very last one in the index.
Not sure I fully understand but it sounds like you want a transaction,
which is not going to happen.

Internally when Cassandra updates a secondary index it does the same
thing. But it synchronises updates around the same row so one thread will
apply the changes at a time.

Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 16/08/2012, at 12:34 PM, "Hiller, Dean"
<Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov><mailto:Dean.Hiller@nrel.gov>>
wrote:

1.  Can playOrm be listed on cassandra's list of ORMs?  It supports a
JQL/HQL query on a trillion rows in under 100ms (partitioning is the
trick so you can JQL a partition)
2.  Many applications have a common indexing problem and I was wondering
if cassandra has or could have any support for this in the futureŠ.

When using wide row indexes, you frequently have
<indexedValue>.<primaryKey> as the composite key.  This means when you
have your object like so in the database

Activity {
pk: 65
name: bill
}

And then two servers want to save it as

Activity {
pk:65
name:tim
}
Activity {
pk:65
name:mike
}

Each server will remove <bill><65> and BOTH servers will add <tim><65>
AND <mike><65> BUT one of them will really be a lie!!!!!  I was wondering
if cassandra has or will ever support eventual constancy where it keeps
both the REMOVE AND the ADD together such until it is on all 3 replicated
nodes and in resolving the consistency would end up with an index that
only has the very last one in the index.

Thanks,
Dean





Mime
View raw message