cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Eineke (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-5922) Delete doesn't delete data.
Date Thu, 22 Aug 2013 17:49:52 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-5922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Eineke updated CASSANDRA-5922:
------------------------------------

    Description: 
In a nutshell, I'm running several test cases against my Cassandra JPA implementation (astyanax-jpa,
see https://github.com/ceineke/astyanax-jpa) and sometimes (!) batched deletes seem not to
delete all rows specified in the batch. 

Here's the sequence of prepared CQL3 statements that is causing the issue to appear:

TRUNCATE compositeentity;

(delete all records so we have a clean slate)

INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree,
astring, auuid) VALUES (?, ?, ?, ?, ?);
INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree,
astring, auuid) VALUES (?, ?, ?, ?, ?);

(insert two unique rows into the table)

SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;

(load both rows from the table to validate their existence)

SELECT COUNT(1) FROM compositeentity;

(counts rows to validate the number of records in the table)

BEGIN BATCH  DELETE  FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo
= ? AND compositekeypartthree = ?; DELETE  FROM compositeentity WHERE compositekeypartone
= ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; APPLY BATCH;

(uses a logged batch to delete the two rows from the table)

SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;

(tries loads the rows from the table to check that they don't exist anymore)

After the delete, Cassandra has deleted only the first row so that the second SELECT here
actually returns data. So far, this behaviour occurs randomly.

This happens even if there's a long sleep (1s, 10s) between the batch delete and the selects.
It is always the second row that isn't deleted, never the first.

Thinking it might be timing issue (based on http://ria101.wordpress.com/2011/02/08/cassandra-the-importance-of-system-clocks-avoiding-oom-and-how-to-escape-oom-meltdown/),
I've set up NTP to keep the clocks synchronized across all nodes (one node acts as the master
which syncs to time.nrc.ca and {0,1,2,3}.ca.pool.ntp.org, whereas the remaining ones sync
against the master. This hasn't reduced the number of times this behaviour crops up.

(I am executing all statements with QUORUM level consistency.)

I'm open to suggestions as to why this occurs and how I can fix it, if this can be fixed.

  was:
In a nutcase, I'm running several test cases against my Cassandra JPA implementation (astyanax-jpa,
see https://github.com/ceineke/astyanax-jpa) and sometimes(!) batched deletes seem not to
delete all data. 

Here's the sequence of prepared CQL3 statements that is causing the issue to appear:

TRUNCATE compositeentity;

(delete all records so we have a clean slate)

INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree,
astring, auuid) VALUES (?, ?, ?, ?, ?);
INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree,
astring, auuid) VALUES (?, ?, ?, ?, ?);

(insert two unique rows into the table)

SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;

(load both rows from the table to validate their existence)

SELECT COUNT(1) FROM compositeentity;

(counts rows to validate the number of records in the table)

BEGIN BATCH  DELETE  FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo
= ? AND compositekeypartthree = ?; DELETE  FROM compositeentity WHERE compositekeypartone
= ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; APPLY BATCH;

(uses a logged batch to delete the two rows from the table)

SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;

(tries loads the rows from the table to check that they don't exist anymore)

After the delete, Cassandra has deleted only the first row so that the second SELECT here
actually returns data. So far, this behaviour occurs randomly.

This happens even if there's a long sleep (1s, 10s) between the batch delete and the selects.
It is always the second row that isn't deleted, never the first.

Thinking it might be timing issue (based on http://ria101.wordpress.com/2011/02/08/cassandra-the-importance-of-system-clocks-avoiding-oom-and-how-to-escape-oom-meltdown/),
I've set up NTP to keep the clocks synchronized across all nodes (one node acts as the master
which syncs to time.nrc.ca and {0,1,2,3}.ca.pool.ntp.org, whereas the remaining ones sync
against the master. This hasn't reduced the number of times this behaviour crops up.

(I am executing all statements with QUORUM level consistency.)

I'm open to suggestions as to why this occurs and how I can fix it, if this can be fixed.

    
> Delete doesn't delete data.
> ---------------------------
>
>                 Key: CASSANDRA-5922
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5922
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 4-node cluster w/ Cassandra v1.2.8
> Oracle JDK 1.6.0_45
> Netflix Astyanax 1.56.42
> Quorum read and write consistency level
>            Reporter: Chris Eineke
>
> In a nutshell, I'm running several test cases against my Cassandra JPA implementation
(astyanax-jpa, see https://github.com/ceineke/astyanax-jpa) and sometimes (!) batched deletes
seem not to delete all rows specified in the batch. 
> Here's the sequence of prepared CQL3 statements that is causing the issue to appear:
> TRUNCATE compositeentity;
> (delete all records so we have a clean slate)
> INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree,
astring, auuid) VALUES (?, ?, ?, ?, ?);
> INSERT INTO compositeentity (compositekeypartone, compositekeyparttwo, compositekeypartthree,
astring, auuid) VALUES (?, ?, ?, ?, ?);
> (insert two unique rows into the table)
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> (load both rows from the table to validate their existence)
> SELECT COUNT(1) FROM compositeentity;
> (counts rows to validate the number of records in the table)
> BEGIN BATCH  DELETE  FROM compositeentity WHERE compositekeypartone = ? AND compositekeyparttwo
= ? AND compositekeypartthree = ?; DELETE  FROM compositeentity WHERE compositekeypartone
= ? AND compositekeyparttwo = ? AND compositekeypartthree = ?; APPLY BATCH;
> (uses a logged batch to delete the two rows from the table)
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> SELECT compositekeypartone, compositekeyparttwo, compositekeypartthree FROM compositeentity
WHERE compositekeypartone = ? AND compositekeyparttwo = ? AND compositekeypartthree = ?;
> (tries loads the rows from the table to check that they don't exist anymore)
> After the delete, Cassandra has deleted only the first row so that the second SELECT
here actually returns data. So far, this behaviour occurs randomly.
> This happens even if there's a long sleep (1s, 10s) between the batch delete and the
selects. It is always the second row that isn't deleted, never the first.
> Thinking it might be timing issue (based on http://ria101.wordpress.com/2011/02/08/cassandra-the-importance-of-system-clocks-avoiding-oom-and-how-to-escape-oom-meltdown/),
I've set up NTP to keep the clocks synchronized across all nodes (one node acts as the master
which syncs to time.nrc.ca and {0,1,2,3}.ca.pool.ntp.org, whereas the remaining ones sync
against the master. This hasn't reduced the number of times this behaviour crops up.
> (I am executing all statements with QUORUM level consistency.)
> I'm open to suggestions as to why this occurs and how I can fix it, if this can be fixed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message