incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: How to bring cluster to consistency
Date Mon, 13 Feb 2012 19:33:07 GMT
> Sorry if this is a 4th copy of letter, but cassandra.apache.org constantly tells me that
my message looks like spam…
Send as text. 

What version are you using ? 
It looks like you are using the ByteOrderedPartitioner , is that correct ? 

I would try to get the repair done first, what was the error ? 

> I believe it is affected by data size. At least some estimation about how much time and
memory it could take would be of use.
You can try to reduce the memory requirements of repair by reducing the in_memory_compaction_limit.
How much memory do you have allocated to cassandra ?

BUT… 1.5T of data is a lot for a single node. Most people work with around 200G to 400G
per node. Otherwise things like repair and move take a very long time. There are also a number
of structures (bloom filters, index samples) maintained in memory that vary with respect to
the data load. 

Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 14/02/2012, at 2:35 AM, Nikolay Kоvshov wrote:

> Sorry if this is a 4th copy of letter, but cassandra.apache.org constantly tells me that
my message looks like spam...
> 
>> 2/ both of your nodes seem to be using the same token? The output indicates that
100% of your key range is assigned to 10.111.1.141 (and therefore 10.111.1.142 holds replicas
only)
> 
> Well, I didn't assign anything. I just filled nodes with data, that's Cassandra itself
who assigned that. I am trying to perform nodetool move now. Still I didn't understand from
wiki what that means (keys assigned to servers). When both servers are up I can write to 1
and read from 2, or I can write to 2 and read from 1 and all works perfect.
> 
>> 3/ maybe repair is being affected by above, but in my experience it can be sensitive
> 
> I believe it is affected by data size. At least some estimation about how much time and
memory it could take would be of use.
> 
> 13.02.2012, 17:19, "Dominic Williams" <dwilliams@fightmymonster.com>:
>> Hi Nikolay,Some points that may be useful:
>> 1/ auto_bootstrap = true is used for telling a new node to join the ring (the cluster).
It has nothing to do with hinted handoff
>> 2/ both of your nodes seem to be using the same token? The output indicates that
100% of your key range is assigned to 10.111.1.141 (and therefore 10.111.1.142 holds replicas
only)
>> 3/ maybe repair is being affected by above, but in my experience it can be sensitive
>> 
>> On 13 February 2012 13:06, Nikolay Kоvshov <nkovshov@yandex.ru> wrote:
>>> Hello everybody
>>> 
>>> I have a very simple cluster containing 2 servers. Replication_factor = 2, Consistency_level
of reads and writes = 1
>>> 
>>> 10.111.1.141    datacenter1 rack1       Up     Normal  1.5 TB          100.00%
vjpigMzv4KkX3x7z
>>> 10.111.1.142    datacenter1 rack1       Up     Normal  1.41 TB         0.00%
  聶jpigMzv4KkX3x7z
>>> 
>>> Note the size please.
>>> 
>>> Say, server1 cassandra dies and I restart it later. Hinted_handoff = enabled,
auto_bootstrap = true
>>> 
>>> During that time server2 received reads and writes. I want changes to be copied
to server1 when it joins the cluster. As I have replication_factor 2, I suppose each data
piece should be stored on both servers. Auto_bootstrapping doesn't seem to work that way -
changed data doesn't migrate.
>>> 
>>> I run nodetool repair and it is always killed by OOM. What else can I do to bring
cluster to consistency?
>>> 
>>> Thank you in advance


Mime
View raw message