This problem was solved by forming a 3-node large-instance cluster. The pause went away. I thought I would try a single node configuration to test the intensive inserts, as you would expect to just work (though may not performing well). It turns out somehow Cassandra likes to have a minimum amount of clustering. If my experiment is correct, maybe this should be mentioned on the wiki; or else the single node configuration should be improved to at least work smoothly albeit slower.

On Wed, Jun 30, 2010 at 3:21 PM, Steve Lihn <stevelihn@gmail.com> wrote:
Jon,
Looks like this compaction process took 360 seconds. What is it and how do I avoid it?

INFO [COMPACTION-POOL:1] 2010-06-30 14:56:08,667 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-129-Data.db'),org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-150-Data.db'),org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-172-Data.db'),org.apache.cassandra.io.SSTableReader(path='/mnt/itops/cdata/TFO/CurrentHolding-194-Data.db')]
 INFO [COMPACTION-POOL:1] 2010-06-30 15:02:09,537 CompactionManager.java (line 326) Compacted to /mnt/itops/cdata/TFO/CurrentHolding-197-Data.db.  1880884459/1880884459 bytes for 6306383 keys.  Time: 360870ms.

On Wed, Jun 30, 2010 at 3:12 PM, Steve Lihn <stevelihn@gmail.com> wrote:
Jon,
I am experimenting writing 8 million rows into Cassandra and also experienced some random timeouts, even with 10-second timeout parameter.
How do I avoid such timeout at all cost? (I.e. At this time, my priority is to finish the end-to-end test. Don't want the program to fail at all.)
This is a one-node server with ms1G and mx2GB. The rest in cassandra.in.sh are default I think.

Steve