cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <>
Subject [jira] Commented: (CASSANDRA-2319) Promote row index
Date Fri, 18 Mar 2011 10:42:29 GMT


Sylvain Lebresne commented on CASSANDRA-2319:

bq. Agreed... my point is simply that the number of columns-per-key and the number of keys
are inversely proportional: if you have more columns-per-key, you have less keys, and vice-versa.
The index will grow proportionally with the total number of columns, not with the number of

I do not share your confidence that this is axiomatic. It is certainly not axiomatic to the
data model. Anyway, that was just a remark, not a criticism of the approach.

bq. Yea... the key cache as it exists does not necessarily need to change, but at some point
we'll want to update it to include the improvements from this ticket.

Maybe there is a misunderstanding here. I assumed that promoting the row index implied removing
the row index (in the favor of a richer sstable index).  And even though a first iteration
of this doesn't necessary imply this removal, I'll still assume it because I believe this
would be weird to keep in the long run even if we keep it in the short run.

So if you don't have a row index, caching row key position as the actual key cache does will
be counter-productive for any non-narrow row, since looking at the sstable index would give
you closer to the column. So it would make the key cache as it exists only useful for narrow
row (which makes it less useful though not useless).

bq. It depends on the number of unique queries to the row, but I'm willing to bet that the
number of unique queries to a row is relatively low.

Take time series (which I doubt can be called a niche use case). If the start of your slice
query depends on the current time, almost all the query will be unique. Or if you page on
the time series and it have a reasonably high rate of inserts, then the pages will be always
changing and thus will be your query.  Given how long it took me to come up with those two
examples (that I did personally used btw, it's not just my imagination running wild), I suspect
there is a number of other similar cases.

Will those be a minority of all the queries on wide rows ? I don't know, probably for some
people but maybe not for others. People come up with new ways to use the Cassandra data model
all the time, let's not base our reflexion on unchecked assumptions of the kind of queries
people do.

Is that a big deal considering that in promoting the row index we will be at 2 seeks for those
case but we're already at 2 seeks on a key cache hit ?  Probably not (though for the pleasure
of nitpicking I'll add that the 2 seeks in the current case of a key cache hit are closer
on disk that how the 2 seeks with the promoted index would be).

Am I willing to not keep those case in mind because Stu is willing to bet it doesn't matter
? Certainly not (which doesn't mean I don't love you).

Anyways, I'm in favor of trying this (honestly more because I believe this could make checksumming
and compression easier to add than anything else).

> Promote row index
> -----------------
>                 Key: CASSANDRA-2319
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Stu Hood
>            Assignee: Stu Hood
>              Labels: index, timeseries
>             Fix For: 0.8
> The row index contains entries for configurably sized blocks of a wide row. For a row
of appreciable size, the row index ends up directing the third seek (1. index, 2. row index,
3. content) to nearby the first column of a scan.
> Since the row index is always used for wide rows, and since it contains information that
tells us whether or not the 3rd seek is necessary (the column range or name we are trying
to slice may not exist in a given sstable), promoting the row index into the sstable index
would allow us to drop the maximum number of seeks for wide rows back to 2, and, more importantly,
would allow sstables to be eliminated using only the index.
> An example usecase that benefits greatly from this change is time series data in wide
rows, where data is appended to the beginning or end of the row. Our existing compaction strategy
gets lucky and clusters the oldest data in the oldest sstables: for queries to recently appended
data, we would be able to eliminate wide rows using only the sstable index, rather than needing
to seek into the data file to determine that it isn't interesting. For narrow rows, this change
would have no effect, as they will not reach the threshold for indexing anyway.
> A first cut design for this change would look very similar to the file format design
proposed on #674: row keys clustered,
column names clustered, and offsets clustered and delta encoded.

This message is automatically generated by JIRA.
For more information on JIRA, see:

View raw message