Txs Sylvain. Interesting feedback
For now higher consistency level is not fault tolerant (two node setup) but worth to consider for other deployments.

Further investigation learns that system has quit some iowaits. It seems that we are pushing it to far.
It is obvious that counter cf's increments are more expensive then regular cf's. But can we expect linear scaling?

Txs!

David

On Fri, Nov 30, 2012 at 1:19 PM, Sylvain Lebresne <sylvain@datastax.com> wrote:
To my understanding client will not be notified because this read step is asynchronously.

Yes, though for the record it's actually synchronous if you use anything else than CL.ONE.
 
Also the other replica cannot be updated properly (replicate-on-write stage is backing up).
Question is:
- Will the counter value eventually become correct?

Yes, provided you either have read repair enabled or run repair at some point. Meaning that if the read gets dropped, there is no hinted hand-off involved so read repair and repair are the only things that will make things eventually consistent (but they will).
 
- Read repair is turned off for the moment, but would this (not reconciliated) memtable  value be merged correctly if read repair would be enabled?

Yes
 
- What with the final value after running nodetool repairs or compaction process...

It'll be fine. 

--
Sylvain