cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dinesh Shanbhag <dinesh.shanb...@isanasystems.com>
Subject Re: Cassandra 3.1 - Aggregation query failure
Date Mon, 21 Dec 2015 09:49:51 GMT

Thanks for the pointers!  I edited jvm.options in 
$CASSANDRA_HOME/conf/jvm.options to increase -Xms and -Xmx to 1536M.  
The result is the same.

And in $CASSANDRA_HOME/logs/system.log, grep GC system.log produces this 
(when jvm.options had not been changed):

INFO  [Service Thread] 2015-12-18 15:26:31,668 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 296ms.  CMS Old Gen: 18133664 -> 15589256; 
Code Cache: 5650880 -> 8122304; Compressed Class Space: 2530064 -> 
3345624; Metaspace: 21314000 -> 28040984; Par Eden Space: 7019256 -> 
164070848;
INFO  [Service Thread] 2015-12-18 15:48:39,736 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 379ms.  CMS Old Gen: 649257416 -> 84190176; 
Code Cache: 20772224 -> 20726848; Par Eden Space: 2191408 -> 52356736; 
Par Survivor Space: 2378448 -> 2346840
INFO  [Service Thread] 2015-12-18 15:58:35,118 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 406ms.  CMS Old Gen: 648847808 -> 86954856; 
Code Cache: 21182080 -> 21188032; Par Eden Space: 1815696 -> 71525744; 
Par Survivor Space: 2388648 -> 2364696
INFO  [Service Thread] 2015-12-18 16:13:45,821 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 211ms.  CMS Old Gen: 648343768 -> 73135720; 
Par Eden Space: 3224880 -> 7957464; Par Survivor Space: 2379912 -> 2414520
INFO  [Service Thread] 2015-12-18 16:32:46,419 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 387ms.  CMS Old Gen: 648476072 -> 68888832; 
Par Eden Space: 2006624 -> 64263360; Par Survivor Space: 2403792 -> 2387664
INFO  [Service Thread] 2015-12-18 16:42:38,648 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 365ms.  CMS Old Gen: 649126336 -> 137359384; 
Code Cache: 22972224 -> 22979840; Metaspace: 41374464 -> 41375104; Par 
Eden Space: 4286080 -> 154449480; Par Survivor Space: 1575440 -> 2310768
INFO  [Service Thread] 2015-12-18 16:51:57,538 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 322ms.  CMS Old Gen: 648338928 -> 79783856; 
Par Eden Space: 2058968 -> 56931312; Par Survivor Space: 2342760 -> 2400336
INFO  [Service Thread] 2015-12-18 17:02:49,543 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 212ms.  CMS Old Gen: 648702008 -> 122954344; 
Par Eden Space: 3269032 -> 61433328; Par Survivor Space: 2395824 -> 3448760
INFO  [Service Thread] 2015-12-18 17:11:54,090 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 306ms.  CMS Old Gen: 648748576 -> 70965096; 
Par Eden Space: 2174840 -> 27074432; Par Survivor Space: 2365992 -> 2373984
INFO  [Service Thread] 2015-12-18 17:22:28,949 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 350ms.  CMS Old Gen: 648243024 -> 90897272; 
Par Eden Space: 2150168 -> 43487192; Par Survivor Space: 2401872 -> 2410728


After modifying jvm.options to increase -Xms & -Xmx (to 1536M):

INFO  [Service Thread] 2015-12-21 11:39:24,918 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 342ms.  CMS Old Gen: 18579136 -> 16305144; 
Code Cache: 8600128 -> 10898752; Compressed Class Space: 3431288 -> 
3761496; Metaspace: 29551832 -> 33307352; Par Eden Space: 4822000 -> 
94853272;
INFO  [Service Thread] 2015-12-21 11:39:30,710 GCInspector.java:284 - 
ParNew GC in 206ms.  CMS Old Gen: 22932208 -> 41454520; Par Eden Space: 
167772160 -> 0; Par Survivor Space: 13144872 -> 20971520
INFO  [Service Thread] 2015-12-21 13:08:14,922 GCInspector.java:284 - 
ConcurrentMarkSweep GC in 468ms.  CMS Old Gen: 21418016 -> 16146528; 
Code Cache: 11693888 -> 11744704; Compressed Class Space: 4331224 -> 
4344192; Metaspace: 37191144 -> 37249960; Par Eden Space: 146089224 -> 
148476848;
INFO  [Service Thread] 2015-12-21 13:08:53,068 GCInspector.java:284 - 
ParNew GC in 216ms.  CMS Old Gen: 16146528 -> 26858568; Par Eden Space: 
167772160 -> 0;


Earlier the node had OpenJDK 8.  For today's tests I installed and used 
Oracle Java 8.

Do the above messages provide any clue? Or any debug logging I can 
enable to progress further?
Thanks,
Dinesh.

On 12/18/2015 9:56 PM, Tyler Hobbs wrote:
>
> On Fri, Dec 18, 2015 at 9:17 AM, DuyHai Doan <doanduyhai@gmail.com 
> <mailto:doanduyhai@gmail.com>> wrote:
>
>     Cassandra will perform a full table scan and fetch all the data in
>     memory to apply the aggregate function.
>
>
> Just to clarify for others on the list: when executing aggregation 
> functions, Cassandra /will/ use paging internally, so at most one page 
> worth of data will be held in memory at a time.  However, if your 
> aggregation function retains a large amount of data, this may 
> contribute to heap pressure.
>
>
> -- 
> Tyler Hobbs
> DataStax <http://datastax.com/>


Mime
View raw message