incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Malone <m...@simplegeo.com>
Subject Re: Ring management and load balance
Date Fri, 26 Mar 2010 21:35:31 GMT
2010/3/26 Roland Hänel <roland@haenel.me>

> Jonathan,
>
> I agree with your idea about a tool that could 'propose' good token choices
> for optimal load-balancing.
>
> If I was going to write such a tool: do you think the thrift API provides
> the necessary information? I think with the RandomPartitioner you cannot
> scan all your rows to actually find out how big certain ranges of rows are.
> And even with the OPP (that is the major target for this kind of tool, for
> sure) you would have to fetch all row's content just to find out how large
> it is, right?
>

With the random partitioner there's no need to suggest a token. The key
space is statistically random so you should be able to just split 2^128 into
equal sized segments and get fairly equal storage load. Your read / write
load could get out of whack if you have hot spots and stuff, I guess. But
for a large distributed data set I think that's unlikely.

For order preserving partitioners it's harder. We've been thinking about
this issue at SimpleGeo and were planning on implementing an algorithm that
could determine the median row key statistically without having to inspect
every key. Basically, it would pull a random sample of row keys (maybe from
the Index file?) and then determine the median of that sample. Thoughts?

Mike

Mime
View raw message