incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: node restart taking too long
Date Tue, 16 Aug 2011 21:53:00 GMT
the logs say it took a long time to read a saved row cache. Try removing the files from the
saved_caches dir as Jonathan suggested. 

The collecting log lines with the INT max count are indicative of the IdentityQueryFilter.
One of the places it is used is when adding rows to the cache. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 16/08/2011, at 11:35 PM, Yan Chunlu wrote:

>  I saw alot slicequeryfilter things if changed the log level to DEBUG.  just thought
even bring up a new node will be faster than start the old one..... it is wired
> 
> DEBUG [main] 2011-08-16 06:32:49,213 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:225@1313068845474382
> DEBUG [main] 2011-08-16 06:32:49,245 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:453@1310999270198313
> DEBUG [main] 2011-08-16 06:32:49,251 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:26@1313199902088827
> DEBUG [main] 2011-08-16 06:32:49,576 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:157@1313097239332314
> DEBUG [main] 2011-08-16 06:32:50,674 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:41729@1313190821826229
> DEBUG [main] 2011-08-16 06:32:50,811 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:6@1313174157301203
> DEBUG [main] 2011-08-16 06:32:50,867 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:98@1312011362250907
> DEBUG [main] 2011-08-16 06:32:50,881 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:42@1313201711997005
> DEBUG [main] 2011-08-16 06:32:50,910 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:96@1312939986190155
> DEBUG [main] 2011-08-16 06:32:50,954 SliceQueryFilter.java (line 123) collecting 0 of
2147483647: 76616c7565:false:621@1313192538616112
> 
> 
> 
> On Tue, Aug 16, 2011 at 7:32 PM, Yan Chunlu <springrider@gmail.com> wrote:
> but it seems the row cache is cluster wide, how will  the change of row cache affect
the read speed?
> 
> 
> On Mon, Aug 15, 2011 at 7:33 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
> Or leave row cache enabled but disable cache saving (and remove the
> one already on disk).
> 
> On Sun, Aug 14, 2011 at 5:05 PM, aaron morton <aaron@thelastpickle.com> wrote:
> >  INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547)
> > completed loading (1744370 ms; 200000 keys) row cache for COMMENT
> >
> > It's taking 29 minutes to load 200,000 rows in the  row cache. Thats a
> > pretty big row cache, I would suggest reducing or disabling it.
> > Background http://www.datastax.com/dev/blog/maximizing-cache-benefit-with-cassandra
> >
> > and server can not afford the load then crashed. after come back, node 3 can
> > not return for more than 96 hours
> >
> > Crashed how ?
> > You may be seeing https://issues.apache.org/jira/browse/CASSANDRA-2280
> > Watch nodetool compactionstats to see when the Merkle tree build finishes
> > and nodetool netstats to see which CF's are streaming.
> > Cheers
> > -----------------
> > Aaron Morton
> > Freelance Cassandra Developer
> > @aaronmorton
> > http://www.thelastpickle.com
> > On 15 Aug 2011, at 04:23, Yan Chunlu wrote:
> >
> >
> > I got 3 nodes and RF=3, when I repairing ndoe3, it seems alot data
> > generated.  and server can not afford the load then crashed.
> > after come back, node 3 can not return for more than 96 hours
> >
> > for 34GB data, the node 2 could restart and back online within 1 hour.
> >
> > I am not sure what's wrong with node3 and should I restart node 3 again?
> > thanks!
> >
> > Address         Status State   Load            Owns    Token
> >
> > 113427455640312821154458202477256070484
> > node1     Up     Normal  34.11 GB        33.33%  0
> > node2     Up     Normal  31.44 GB        33.33%
> > 56713727820156410577229101238628035242
> > node3     Down   Normal  177.55 GB       33.33%
> > 113427455640312821154458202477256070484
> >
> >
> > the log shows it is still going on, not sure why it is so slow:
> >
> >
> >  INFO [main] 2011-08-14 08:55:47,734 SSTableReader.java (line 154) Opening
> > /cassandra/data/COMMENT
> >  INFO [main] 2011-08-14 08:55:47,828 ColumnFamilyStore.java (line 275)
> > reading saved cache /cassandra/saved_caches/COMMENT-RowCache
> >  INFO [main] 2011-08-14 09:24:52,198 ColumnFamilyStore.java (line 547)
> > completed loading (1744370 ms; 200000 keys) row cache for COMMENT
> >  INFO [main] 2011-08-14 09:24:52,299 ColumnFamilyStore.java (line 275)
> > reading saved cache /cassandra/saved_caches/COMMENT-RowCache
> >  INFO [CompactionExecutor:1] 2011-08-14 10:24:55,480 CacheWriter.java (line
> > 96) Saved COMMENT-RowCache (200000 items) in 2535 ms
> >
> >
> >
> >
> >
> >
> 
> 
> 
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
> 
> 


Mime
View raw message