There is a limit on the size of the commit log and on how long Hints are stored for. 

I'm not sure why your load was different, I think it was left of hints and commit log. But it's not always easy to diagnose thingsvia email. 

Hopefully nodetool drain or deleting the rest system and starting again will get you moving forwards again.

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton

On 14/12/2012, at 12:50 AM, Sergey Olefir <solf.lists@gmail.com> wrote:

I'll try nodetool drain, thanks.

But more generally -- are you basically saying that I should not worry about
these things? Data will not keep accumulating indefinitely in production and
it'll not affect performance negatively (despite vast differences in node
load)?

Best regards,
Sergey


aaron morton wrote
try nodetool drain. It will flush everything to disk and the commit log
will be truncated.

HH can be ignored. If you really want them gone they can be purged using
the JMX interface, or you can stop the node and delete the sstables.


Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 13/12/2012, at 10:35 AM, Sergey Olefir &lt;

solf.lists@

&gt; wrote:

Nick Bailey-2 wrote
Dropping a keyspace causes a snapshot to be taken of the keyspace before
it
is removed from the schema. So it won't actually delete any data. You
can
manually delete the data from /var/lib/cassandra/

<ks>
/&lt;cf[s]&gt;/snapshots

Indeed, it looks like snapshot is on the file system. However it looks
like
it is not the only thing by a long shot, i.e.:
cassa1-1:/var/log/cassandra# du -k /spool1/cassandra/data/1.1/
375372
/spool1/cassandra/data/1.1/rainmanLoadTestKeyspace/marquisColumnFamily/snapshots/1355222054452-marquisColumnFamily
375376
/spool1/cassandra/data/1.1/rainmanLoadTestKeyspace/marquisColumnFamily/snapshots
375380
/spool1/cassandra/data/1.1/rainmanLoadTestKeyspace/marquisColumnFamily
375384  /spool1/cassandra/data/1.1/rainmanLoadTestKeyspace
4       /spool1/cassandra/data/1.1/system/Versions
52      /spool1/cassandra/data/1.1/system/schema_columns
4       /spool1/cassandra/data/1.1/system/Schema
28      /spool1/cassandra/data/1.1/system/NodeIdInfo
4       /spool1/cassandra/data/1.1/system/Migrations
28      /spool1/cassandra/data/1.1/system/schema_keyspaces
28      /spool1/cassandra/data/1.1/system/schema_columnfamilies
786348  /spool1/cassandra/data/1.1/system/HintsColumnFamily
52      /spool1/cassandra/data/1.1/system/LocationInfo
4       /spool1/cassandra/data/1.1/system/IndexInfo
786556  /spool1/cassandra/data/1.1/system
1161944 /spool1/cassandra/data/1.1/


And also 700+MB in the commitlog. Neither of which seemed to 'go away' on
its own when idle or even after running nodetool repair/cleanup and even
dropping keyspace.

I suppose these hints and commitlog may be the reason behind huge
difference
in load on nodes -- but why does it happen and more importantly is it
harmful? Will it keep accumulating?



--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Multiple-Data-Center-shows-very-uneven-load-tp7584197p7584256.html
Sent from the

cassandra-user@.apache

mailing list archive at Nabble.com.





--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Multiple-Data-Center-shows-very-uneven-load-tp7584197p7584264.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.