cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Günter Ladwig <guenter.lad...@kit.edu>
Subject Re: Storing single rows on multiple nodes
Date Sat, 09 Jul 2011 21:16:11 GMT
Hi,

On 09.07.2011, at 23:37, Dan Kuebrich wrote:

> Perhaps I misunderstand your proposal, but it seems that even with your manual key placement
schemes, the row would still be huge, no matter what node it gets placed on.  A better solution
might be figuring out how to make each row into a few smaller ones to get better balancing
of load and also faster reads.

I probably could have been more clear. The idea is to randomly choose one node among several
for a single key each time some data is added to a row. Say, a particular key is normally
assigned to node n. Then, for each write to that key, we randomly choose one of the nodes
n, n+1, n+2, ..., n+k to write the data (or we could choose the node with the least load).
Of course, if one wants to read all data for that key, all these nodes have to be queried,
because each node will store a chunk of the data for the key.

> - Can you segment the column(s) of the row into different, predictably-named rows?

Yes and no. We actually do not know in advance which of the rows will be the ones that grow
that large. However, and this is the second solution I described, it would be possible to
randomly choose some suffix that is added when some data is added to a key. For example, we
might have a key "abc" and a predefined list of suffixes (1, 2, 3). When adding data instead
of writing to key "abc", we randomly choose one of the suffixes and then, for example, write
to "abc1". Of course, the number of suffixes determines how high the probability is that the
modified keys will actually be stored on different nodes.

> - Or segment into different rows and use a secondary index to find the rows that are
part of a particular RDF?

This is actually something I hadn't looked at yet, thanks for the pointer!

> - And/or compress the RDF data (maybe you're already doing that) to reduce the impact
of large rows?

While compression would certainly help, it does not really change the underlying problem,
just delay its effect. Actually, I hope that Cassandra itself will at some point take care
of compression ;) Another problem is that you can't actually just increase cluster size to
scale to larger datasets, because the constraint is the disk space on single nodes.

Cheers,
Günter

> 
> On Sat, Jul 9, 2011 at 4:27 PM, Günter Ladwig <guenter.ladwig@kit.edu> wrote:
> Hi all,
> 
> we are currently looking at using Cassandra to store highly skewed RDF data. With the
indexes we use it may happen that a single row contains up to 20% of the whole dataset, meaning
that it can grow larger than available disk space on single nodes. In [1], it says that this
limitation is not likely to change in the future, but I was wondering if anybody has looked
at this problem?
> 
> One thing that comes to mind is a simple approach to DHT load-balancing [2], where keys
are assigned to one node of several random alternatives (which means that for reading, all
these nodes have to be queried). This is a bit similar to replication, except, of course,
that only one copy of the data is stored. As this would require changes to the Cassandra code
base, we could "simulate" this by randomly choosing one of several predefined suffixes and
appending it to a key before storing it. By modifying a key this way, we could be somewhat
sure that it will be stored at a different node. The first solution would certainly be preferable.
> 
> Any thoughts or experiences? Failing that, maybe someone can give me a pointer into the
Cassandra code base, where something like the [2] should be implemented.
> 
> Cheers,
> Günter
> 
> [1] http://wiki.apache.org/cassandra/CassandraLimitations
> [2] Byers at el.: Simple Load Balancing for Distributed Hash Tables, http://www.springerlink.com/content/r9r4qcqxc2bmfqmr/
> 
> --
> 
> Dipl.-Inform. Günter Ladwig
> 
> Karlsruhe Institute of Technology (KIT)
> Institute AIFB
> 
> Englerstraße 11 (Building 11.40, Room 250)
> 76131 Karlsruhe, Germany
> Phone: +49 721 608-47946
> Email: guenter.ladwig@kit.edu
> Web: www.aifb.kit.edu
> 
> KIT – University of the State of Baden-Württemberg and National Large-scale Research
Center of the Helmholtz Association
> 
> 

--  

Dipl.-Inform. Günter Ladwig

Karlsruhe Institute of Technology (KIT)
Institute AIFB

Englerstraße 11 (Building 11.40, Room 250)
76131 Karlsruhe, Germany
Phone: +49 721 608-47946
Email: guenter.ladwig@kit.edu
Web: www.aifb.kit.edu

KIT – University of the State of Baden-Württemberg and National Large-scale Research Center
of the Helmholtz Association


Mime
View raw message