Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of patrik.modesto@gmail.com
 designates 209.85.215.44 as permitted sender)
MIME-Version: 1.0
From: Patrik Modesto <patrik.modesto@gmail.com>
Date: Tue, 6 Dec 2011 10:50:17 +0100
Message-ID: 
 <CAC43XBn7MvOo5yXNWW--WPgi_Az=8xNrxiwOi_GUK40d99Co0A@mail.gmail.com>
Subject: Cassandra not suitable?
To: user@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8

Hi,

I'm quite desperate about Cassandra's performance in our production
cluster. We have 8 real-HW nodes, 32core CPU, 32GB memory, 4 disks in
raid10, cassandra 0.8.8, RF=3 and Hadoop.
We four keyspaces, one is the large one, it has 2 CFs, one is kind of
index, the other holds data. There are about 7milinon rows, mean row
size is 7kB. We run several mapreduce tasks, most of them just reads
from cassandra and writes to hdfs, but one fetch rows from cassnadra,
compute something and write it back, for each row we compute three new
json values, about 1kB each (they get overwritten next round).

We got lots and lots of Timeout exceptions, LiveSSTablesCount over
100. Reapir doesn't finish even in 24hours, reading from the other
keyspaces timeouts as well.  We set compaction_throughput_mb_per_sec:
0 but it didn't help.

Did we choose wrong DB for our usecase?

Regards,
Patrik

This is from one node:

 INFO 10:28:40,035 Pool Name                    Active   Pending   Blocked
 INFO 10:28:40,036 ReadStage                        96       695         0
 INFO 10:28:40,037 RequestResponseStage              0         0         0
 INFO 10:28:40,037 ReadRepairStage                   0         0         0
 INFO 10:28:40,037 MutationStage                     1         1         0
 INFO 10:28:40,038 ReplicateOnWriteStage             0         0         0
 INFO 10:28:40,038 GossipStage                       0         0         0
 INFO 10:28:40,038 AntiEntropyStage                  0         0         0
 INFO 10:28:40,039 MigrationStage                    0         0         0
 INFO 10:28:40,039 StreamStage                       0         0         0
 INFO 10:28:40,040 MemtablePostFlusher               0         0         0
 INFO 10:28:40,040 FlushWriter                       0         0         0
 INFO 10:28:40,040 MiscStage                         0         0         0
 INFO 10:28:40,041 FlushSorter                       0         0         0
 INFO 10:28:40,041 InternalResponseStage             0         0         0
 INFO 10:28:40,041 HintedHandoff                     1         5         0
 INFO 10:28:40,042 CompactionManager               n/a        27
 INFO 10:28:40,042 MessagingService                n/a   0,16559

And here is the nodetool ring  output:

10.2.54.91      NG          RAC1        Up     Normal  118.04 GB
12.50%  0
10.2.54.92      NG          RAC1        Up     Normal  102.74 GB
12.50%  21267647932558653966460912964485513216
10.2.54.93      NG          RAC1        Up     Normal  76.95 GB
12.50%  42535295865117307932921825928971026432
10.2.54.94      NG          RAC1        Up     Normal  56.97 GB
12.50%  63802943797675961899382738893456539648
10.2.54.95      NG          RAC1        Up     Normal  75.55 GB
12.50%  85070591730234615865843651857942052864
10.2.54.96      NG          RAC1        Up     Normal  102.57 GB
12.50%  106338239662793269832304564822427566080
10.2.54.97      NG          RAC1        Up     Normal  68.03 GB
12.50%  127605887595351923798765477786913079296
10.2.54.98      NG          RAC1        Up     Normal  194.6 GB
12.50%  148873535527910577765226390751398592512