cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: How do secondary indices work
Date Wed, 09 Feb 2011 16:52:38 GMT
"Iterating through all of the rows matching an index clause on your
cluster is guaranteed to touch N/RF of the nodes in your cluster,
because each node only knows about data that is indexed locally."

On Wed, Feb 9, 2011 at 9:13 AM,  <altanis@ceid.upatras.gr> wrote:
> One more question: does each node keep an index of their own values, or is
> the index global?
>
> Alexander
>
>> Thank you very much, this is the information I was looking for. I started
>> adding secondary index functionality to Cassandra myself, and it turns out
>> I am doing almost exactly the same thing. I will try to change my code to
>> use your implementation as well to compare results.
>>
>> Alexander
>>
>>> Alexander:
>>>
>>> The secondary indexes in 0.7.0 (type KEYS) are stored internally in a
>>> column
>>> family, and are kept synchronized with the base data via locking on a
>>> local
>>> node, meaning they are always consistent on the local node. Eventual
>>> consistency still applies between nodes, but a returned result will
>>> always
>>> match your query.
>>>
>>> This index column family stores a mapping from index values to a sorted
>>> list
>>> of matching row keys. When you query for rows between x and y matching a
>>> value z (via the get_indexed_slices call), Cassandra performs a lookup
>>> to
>>> the index column family for the slice of columns in row z between x and
>>> y.
>>> If any matches are found in the index, they are row keys that match the
>>> index clause, and we query the base data to return you those rows.
>>>
>>> Iterating through all of the rows matching an index clause on your
>>> cluster
>>> is guaranteed to touch N/RF of the nodes in your cluster, because each
>>> node
>>> only knows about data that is indexed locally.
>>>
>>> Some portions of the indexing implementation are not fully baked yet:
>>> for
>>> instance, although the API allows you to specify multiple columns, only
>>> one
>>> index will actually be used per query, and the rest of the clauses will
>>> be
>>> brute forced.
>>>
>>> A second secondary index implementation has been on the back burner for
>>> a
>>> while: it provides an identical API, but does not use a column family to
>>> store the index, and should be more efficient for append only data. See
>>> https://issues.apache.org/jira/browse/CASSANDRA-1472
>>>
>>> Thanks,
>>> Stu
>>>
>>> On Wed, Feb 9, 2011 at 2:35 AM, <altanis@ceid.upatras.gr> wrote:
>>>
>>>> Thank you for the links, I did read a bit in the comments of the
>>>> ticket,
>>>> but I couldn't get much out of it.
>>>>
>>>> I am mainly interested in how the index is stored and partitioned, not
>>>> how
>>>> it is used. I think the people in the dev list will probably be better
>>>> qualified to answer that. My questions always seem to get moved to the
>>>> user list, and usually with good cause, but I think this time it should
>>>> be
>>>> in the dev list :) Please move it back, if you can.
>>>>
>>>> Alexander
>>>>
>>>> > AFAIK this was the ticket the original work was done under
>>>> > https://issues.apache.org/jira/browse/CASSANDRA-1415
>>>> >
>>>> > also  http://www.datastax.com/docs/0.7/data_model/secondary_indexes
>>>> > and  http://pycassa.githubcom/pycassa/tutorial.html#indexes may help
>>>> >
>>>> > (sorry on reflection the email prob did not need to be moved from
>>>> dev,
>>>> my
>>>> > bad)
>>>> > Aaron
>>>> >
>>>> > On 09 Feb, 2011,at 09:16 AM, Aaron Morton <aaron@thelastpickle.com>
>>>> wrote:
>>>> >
>>>> > Moving to the user group.
>>>> >
>>>> >
>>>> >
>>>> > On 08 Feb, 2011,at 11:39 PM, altanis@ceid.upatras.gr wrote:
>>>> >
>>>> > Hello,
>>>> >
>>>> > I'd like some information about how secondary indices work under the
>>>> hood.
>>>> >
>>>> > 1) Is data stored in some external data structure, or is it stored in
>>>> an
>>>> > actual Cassandra table, as columns within column families?
>>>> > 2) Is data stored sorted or not? How is it partitioned?
>>>> > 3) How can I access index data?
>>>> >
>>>> > Thanks in a advance,
>>>> >
>>>> > Alexander Altanis
>>>> >
>>>>
>>>
>>
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message