Can you paste you gc config? Also can you take a heap dump at 2 diff points so that we can compare it?
Quick thing to do would be to do a histo live at 2 points and compare
Sent from my iPhone
> INFO [ScheduledTasks:1] 2013-04-15 14:00:02,749 GCInspector.java (line 122) GC for ParNew: 338798 ms for 1 collections, 592212416 used; max is 1046937600
This says GC for New Generation took so long. And this is usually unlikely.
The only situation I am aware of is when a fairly large object is created, and which can not be promoted to Old Generation because it requires such a large *contiguous* memory space that is unavailable at the point in time. This is called promotion failure. So it has to wait until concurrent collector collects a large enough space. Thus you experience stop the world. But I think it is not stop the world, but only stop the new world.
For example in case of Cassandra, a large number of in_memory_compaction_limit_in_mb can cause this. This is a limit when a compaction compacts(merges) rows of a key into the latest in memory. So this creates a large byte array up to the number.
You can confirm this by enabling promotion failure GC logging in the future, and by checking compactions executed at that point in time.