cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: node restart taking too long
Date Sun, 14 Aug 2011 23:33:02 GMT
Or leave row cache enabled but disable cache saving (and remove the
one already on disk).

On Sun, Aug 14, 2011 at 5:05 PM, aaron morton <aaron@thelastpickle.com> wrote:
>  INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547)
> completed loading (1744370 ms; 200000 keys) row cache for COMMENT
>
> It's taking 29 minutes to load 200,000 rows in the  row cache. Thats a
> pretty big row cache, I would suggest reducing or disabling it.
> Background http://www.datastax.com/dev/blog/maximizing-cache-benefit-with-cassandra
>
> and server can not afford the load then crashed. after come back, node 3 can
> not return for more than 96 hours
>
> Crashed how ?
> You may be seeing https://issues.apache.org/jira/browse/CASSANDRA-2280
> Watch nodetool compactionstats to see when the Merkle tree build finishes
> and nodetool netstats to see which CF's are streaming.
> Cheers
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> On 15 Aug 2011, at 04:23, Yan Chunlu wrote:
>
>
> I got 3 nodes and RF=3, when I repairing ndoe3, it seems alot data
> generated.  and server can not afford the load then crashed.
> after come back, node 3 can not return for more than 96 hours
>
> for 34GB data, the node 2 could restart and back online within 1 hour.
>
> I am not sure what's wrong with node3 and should I restart node 3 again?
> thanks!
>
> Address         Status State   Load            Owns    Token
>
> 113427455640312821154458202477256070484
> node1     Up     Normal  34.11 GB        33.33%  0
> node2     Up     Normal  31.44 GB        33.33%
> 56713727820156410577229101238628035242
> node3     Down   Normal  177.55 GB       33.33%
> 113427455640312821154458202477256070484
>
>
> the log shows it is still going on, not sure why it is so slow:
>
>
>  INFO [main] 2011-08-14 08:55:47,734 SSTableReader.java (line 154) Opening
> /cassandra/data/COMMENT
>  INFO [main] 2011-08-14 08:55:47,828 ColumnFamilyStore.java (line 275)
> reading saved cache /cassandra/saved_caches/COMMENT-RowCache
>  INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547)
> completed loading (1744370 ms; 200000 keys) row cache for COMMENT
>  INFO [main] 2011-08-14 09:24:52,299 ColumnFamilyStore.java (line 275)
> reading saved cache /cassandra/saved_caches/COMMENT-RowCache
>  INFO [CompactionExecutor:1] 2011-08-14 10:24:55,480 CacheWriter.java (line
> 96) Saved COMMENT-RowCache (200000 items) in 2535 ms
>
>
>
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message