incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Multiple Data Center shows very uneven load
Date Fri, 14 Dec 2012 01:43:52 GMT
There is a limit on the size of the commit log and on how long Hints are stored for. 

I'm not sure why your load was different, I think it was left of hints and commit log. But
it's not always easy to diagnose thingsvia email. 

Hopefully nodetool drain or deleting the rest system and starting again will get you moving
forwards again.

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 14/12/2012, at 12:50 AM, Sergey Olefir <solf.lists@gmail.com> wrote:

> I'll try nodetool drain, thanks.
> 
> But more generally -- are you basically saying that I should not worry about
> these things? Data will not keep accumulating indefinitely in production and
> it'll not affect performance negatively (despite vast differences in node
> load)?
> 
> Best regards,
> Sergey
> 
> 
> aaron morton wrote
>> try nodetool drain. It will flush everything to disk and the commit log
>> will be truncated.
>> 
>> HH can be ignored. If you really want them gone they can be purged using
>> the JMX interface, or you can stop the node and delete the sstables. 
>> 
>> 
>> Cheers
>> 
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> New Zealand
>> 
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 13/12/2012, at 10:35 AM, Sergey Olefir &lt;
> 
>> solf.lists@
> 
>> &gt; wrote:
>> 
>>> Nick Bailey-2 wrote
>>>> Dropping a keyspace causes a snapshot to be taken of the keyspace before
>>>> it
>>>> is removed from the schema. So it won't actually delete any data. You
>>>> can
>>>> manually delete the data from /var/lib/cassandra/
>>>> 
>> <ks>
>>>> /&lt;cf[s]&gt;/snapshots
>>> 
>>> Indeed, it looks like snapshot is on the file system. However it looks
>>> like
>>> it is not the only thing by a long shot, i.e.:
>>> cassa1-1:/var/log/cassandra# du -k /spool1/cassandra/data/1.1/
>>> 375372 
>>> /spool1/cassandra/data/1.1/rainmanLoadTestKeyspace/marquisColumnFamily/snapshots/1355222054452-marquisColumnFamily
>>> 375376 
>>> /spool1/cassandra/data/1.1/rainmanLoadTestKeyspace/marquisColumnFamily/snapshots
>>> 375380 
>>> /spool1/cassandra/data/1.1/rainmanLoadTestKeyspace/marquisColumnFamily
>>> 375384  /spool1/cassandra/data/1.1/rainmanLoadTestKeyspace
>>> 4       /spool1/cassandra/data/1.1/system/Versions
>>> 52      /spool1/cassandra/data/1.1/system/schema_columns
>>> 4       /spool1/cassandra/data/1.1/system/Schema
>>> 28      /spool1/cassandra/data/1.1/system/NodeIdInfo
>>> 4       /spool1/cassandra/data/1.1/system/Migrations
>>> 28      /spool1/cassandra/data/1.1/system/schema_keyspaces
>>> 28      /spool1/cassandra/data/1.1/system/schema_columnfamilies
>>> 786348  /spool1/cassandra/data/1.1/system/HintsColumnFamily
>>> 52      /spool1/cassandra/data/1.1/system/LocationInfo
>>> 4       /spool1/cassandra/data/1.1/system/IndexInfo
>>> 786556  /spool1/cassandra/data/1.1/system
>>> 1161944 /spool1/cassandra/data/1.1/
>>> 
>>> 
>>> And also 700+MB in the commitlog. Neither of which seemed to 'go away' on
>>> its own when idle or even after running nodetool repair/cleanup and even
>>> dropping keyspace.
>>> 
>>> I suppose these hints and commitlog may be the reason behind huge
>>> difference
>>> in load on nodes -- but why does it happen and more importantly is it
>>> harmful? Will it keep accumulating?
>>> 
>>> 
>>> 
>>> --
>>> View this message in context:
>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Multiple-Data-Center-shows-very-uneven-load-tp7584197p7584256.html
>>> Sent from the 
> 
>> cassandra-user@.apache
> 
>> mailing list archive at Nabble.com.
> 
> 
> 
> 
> 
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Multiple-Data-Center-shows-very-uneven-load-tp7584197p7584264.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.


Mime
View raw message