Check, I understand. Thanks!

The cluster certainly was overloaded and I did not realize that truncate does not tombstone or have a timestamp. Some 'feature' for future implementation, maybe?
It seems odd if you expect the same behaviour of "delete from usertable" (in SQL, not yet in CQL, I presume), especially because the truncate is synced over all nodes before it returns to client, so a truncate may rightfully discard its handoffs, right?
btw, it was very hard to replicate this behaviour, seems to be a rare occurrence...

I wonder though that you don't know YCSB, what do you use to do performance testing? wrote your own or use another tool? In the latter I would like to know what you use :)


Peter Dijkshoorn
Adyen - Payments Made Easy

Visiting address:    	        Mail Address:             
Simon Carmiggeltstraat 6-50    	P.O. Box 10095
1011 DJ Amsterdam               1001 EB Amsterdam
The Netherlands                 The Netherlands

Office +

On 05/07/2012 12:59 PM, aaron morton wrote:
I don't know the YCSB code, but one theory would be…

1) The cluster is overloaded by the test. 
2) A write at CL ALL fails because a node does not respond in time. 
3) The coordinator stores the hint and returns failure to the client. 
4) The client gets an UnavailableException and retries the operation. 

Did the nodes show any dropped messages ? Either in nodetool tpstats or in the logs?

Truncate is meta data operation, unlike deleting columns or rows. When a column is deleted a Tombstone column is written, when row is deleted information is associated with the key, in the context of the CF. Truncate snapshots and then deletes the SSTables on disk, it does not write to the SSTables. So it is possible for a write to be stored with a lower timestamp than the truncate, because truncate does not have a timestamp. 


Aaron Morton
Freelance Developer

On 4/05/2012, at 1:28 AM, Peter Dijkshoorn wrote:

Hi guys,

I got a weird thingy popping up twice today, I run a test where I insert
a milion records via YCSB and edited it to allow me to adjust the
consistency level: the write operations are done with ConsistencyLevel.ALL.
This is send to a 4 (virtual) node cluster with a keyspace 'test' set up
with replication factor 3.
Now I expect that because of the ConsistencyLevel.ALL there is no hinted
handoff active, since writes are to be accepted by all nodes before the
operation returns to the client. The client gets only OK back, none fails.
After the test I run a truncate, and a count which reveals still active
records, time does not matter, I have to re-invoke the truncate to
remove the records.

[cqlsh 2.0.0 | Cassandra 1.0.8 | CQL spec 2.0.0 | Thrift protocol 19.20.0]
cqlsh> use test;
cqlsh:test> truncate usertable;
cqlsh:test> select count(*) from usertable ;

On the cassandra output (-f) I can see that there is some handoff-ing
active, which I did not expect.

Has anyone an idea why the handoff is active while issuing opperations
with ConsistencyLevel.ALL?
Why is the truncate not correctly put in sync and allows subsequent
handoff's delivered of records originally set before the truncate?

Thanks if you can clarify these thing, I did not expect this at all.



Peter Dijkshoorn
Adyen - Payments Made Easy

Visiting address:            Mail Address:             
Simon Carmiggeltstraat 6-50     P.O. Box 10095
1011 DJ Amsterdam               1001 EB Amsterdam
The Netherlands                 The Netherlands

Office +