cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: node restart taking too long
Date Sun, 14 Aug 2011 22:05:17 GMT
>  INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547) completed loading
(1744370 ms; 200000 keys) row cache for COMMENT
It's taking 29 minutes to load 200,000 rows in the  row cache. Thats a pretty big row cache,
I would suggest reducing or disabling it. 
Background http://www.datastax.com/dev/blog/maximizing-cache-benefit-with-cassandra

> and server can not afford the load then crashed. after come back, node 3 can not return
for more than 96 hours
Crashed how ?

You may be seeing https://issues.apache.org/jira/browse/CASSANDRA-2280 
Watch nodetool compactionstats to see when the Merkle tree build finishes and nodetool netstats
to see which CF's are streaming. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 15 Aug 2011, at 04:23, Yan Chunlu wrote:

> 
> 
> I got 3 nodes and RF=3, when I repairing ndoe3, it seems alot data generated.  and server
can not afford the load then crashed.
> after come back, node 3 can not return for more than 96 hours
> 
> for 34GB data, the node 2 could restart and back online within 1 hour.
> 
> I am not sure what's wrong with node3 and should I restart node 3 again? thanks!
> 
> Address         Status State   Load            Owns    Token
>                                                        113427455640312821154458202477256070484
> node1     Up     Normal  34.11 GB        33.33%  0
> node2     Up     Normal  31.44 GB        33.33%  56713727820156410577229101238628035242
> node3     Down   Normal  177.55 GB       33.33%  113427455640312821154458202477256070484
> 
> 
> the log shows it is still going on, not sure why it is so slow:
> 
> 
>  INFO [main] 2011-08-14 08:55:47,734 SSTableReader.java (line 154) Opening /cassandra/data/COMMENT
>  INFO [main] 2011-08-14 08:55:47,828 ColumnFamilyStore.java (line 275) reading saved
cache /cassandra/saved_caches/COMMENT-RowCache
>  INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547) completed loading
(1744370 ms; 200000 keys) row cache for COMMENT
>  INFO [main] 2011-08-14 09:24:52,299 ColumnFamilyStore.java (line 275) reading saved
cache /cassandra/saved_caches/COMMENT-RowCache
>  INFO [CompactionExecutor:1] 2011-08-14 10:24:55,480 CacheWriter.java (line 96) Saved
COMMENT-RowCache (200000 items) in 2535 ms
> 
> 
> 
> 


Mime
View raw message