You do seem to be experiencing a problem indeed, but hard to say what it is from just this (could be anything from a configuration problem, a cassandra bug or a bug in the java driver).
But since you seem to be able to reproduce easily, if you can provide a script that reproduce that issue, I'd gladly have a look (feel free to send it to me privately if you prefer).  

--
Sylvain


On Tue, Mar 5, 2013 at 4:40 PM, Gabriel Ciuloaica <gciuloaica@gmail.com> wrote:
So, I have added more logging to the test app (comments inline). For some reason I'm loosing updates.  In a for loop I'm executing upload, read writetime, download blob. Executed 10 times... See iteration number 2 and 3....

1. initialize session

0    [main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /10.11.1.109 added
1    [main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /10.11.1.200 added



COORDINATOR: 10.11.1.108
UPLOAD LENGTH:214 bytes      --> uploading a blob (INSERT)
UPLOAD:1154 ms
Write time: 1362497519584000 ms --> reading writetime(avatar)
DOWNLOAD LENGTH: 214 bytes   --> download the blob (SELECT)
DOWNLOAD: 134 ms
---> md5 of the blob on upload                                                             ---> md5 of the blob on download
Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 ===  Download Digest MD5 : b1944c41a25192520d33d15d00db2718
 -------------------------------- 0
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:4031 bytes
UPLOAD:675 ms
Write time: 1362497521493000 ms
DOWNLOAD LENGTH: 4031 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 ===  Download Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295
 -------------------------------- 1
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:3392 bytes
UPLOAD:668 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 ===  Download Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585
 -------------------------------- 2
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:253 bytes
UPLOAD:668 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 ===  Download Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585
 -------------------------------- 3
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:266 bytes
UPLOAD:704 ms
Write time: 1362497556815000 ms
DOWNLOAD LENGTH: 3392 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 ===  Download Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585
 -------------------------------- 4
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:3082 bytes
UPLOAD:901 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 ===  Download Digest MD5 : fa2ea1972992cafea1c71b6c3e718058
 -------------------------------- 5
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:1481 bytes
UPLOAD:703 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 135 ms
Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf ===  Download Digest MD5 : fa2ea1972992cafea1c71b6c3e718058
 -------------------------------- 6
COORDINATOR: 10.11.1.109
UPLOAD LENGTH:5214 bytes
UPLOAD:801 ms
Write time: 1362497562076000 ms
DOWNLOAD LENGTH: 3082 bytes
DOWNLOAD: 134 ms
Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 ===  Download Digest MD5 : fa2ea1972992cafea1c71b6c3e718058
 -------------------------------- 7
COORDINATOR: 10.11.1.200
UPLOAD LENGTH:2992 bytes
UPLOAD:665 ms
Write time: 1362497567779000 ms
DOWNLOAD LENGTH: 2992 bytes
DOWNLOAD: 134 ms
Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 ===  Download Digest MD5 : 0848513c1b4214adf73c6ea5509ec294
 -------------------------------- 8
COORDINATOR: 10.11.1.108
UPLOAD LENGTH:3670 bytes
UPLOAD:672 ms
Write time: 1362497567779000 ms
DOWNLOAD LENGTH: 2992 bytes
DOWNLOAD: 136 ms
Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b ===  Download Digest MD5 : 0848513c1b4214adf73c6ea5509ec294
 -------------------------------- 9


Thanks,
Gabi

On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote:
Hi Sylvain,

thanks for fast answer. I have updated keyspace definition and cassandra-topologies.properties to all 3 nodes and restarted each node. Both problems are still reproducible. I'm not able to read my writes and also the selects shows same data as in my previous email.

for write and read I'm using:
private static final String WRITE_STATEMENT = "INSERT INTO avatars (id, image_type, avatar) VALUES (?,?,?);";
private static final String READ_STATEMENT = "SELECT avatar, image_type FROM avatars WHERE id=?";

I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls.

Write snippet:

Session session;
        try {
            session = cassandraSession.getSession();
            BoundStatement stmt = session.prepare(WRITE_STATEMENT)
                    .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
            stmt.enableTracing();
            stmt.setLong("id", accountId);
            stmt.setString("image_type", image.getType());
            stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes()));
            ResultSet result = session.execute(stmt);
            LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace()
                    .getCoordinator().getCanonicalHostName());

        } catch (NoHostAvailableException e) {
            LOG.error("Could not prepare the statement.", e);
            throw new StorageUnavailableException(e);
        } finally {
            cassandraSession.releaseSession();
        }

Read snippet:

Session session = null;
        byte[] imageBytes = null;
        String imageType = "png";
        try {
            session = cassandraSession.getSession();
            BoundStatement stmt = session.prepare(READ_STATEMENT)
                    .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
            stmt.setLong("id", accountId);
            ResultSet result = session.execute(stmt);
            Iterator<Row> it = result.iterator();
            ByteBuffer avatar = null;

            while (it.hasNext()) {
                Row row = it.next();
                avatar = row.getBytes("avatar");
                imageType = row.getString("image_type");
            }
            if (avatar == null) {
                throw new AvatarNotFoundException("Avatar hasn't been found");
            }
            int length = avatar.remaining();
            imageBytes = new byte[length];
            avatar.get(imageBytes, 0, length);

        } catch (NoHostAvailableException e) {
            LOG.error("Could not prepare the statement.", e);
            throw new StorageUnavailableException(e);
        } finally {
            cassandraSession.releaseSession();
        }

Let me know what other information is need it.

Thanks,
Gabi


On 3/5/13 12:52 PM, Sylvain Lebresne wrote:
Without looking into details too closely, I'd say you're probably hitting https://issues.apache.org/jira/browse/CASSANDRA-5292 (since you use NTS+propertyFileSnitch+a DC name in caps).

Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as dc-toronto, which then probably don't match what you have in you property file. This will be fixed in 1.2.3. In the meantime, a workaround would be to use the cassandra-cli to create/update your keyspace definition.

--
Sylvain


On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica <gciuloaica@gmail.com> wrote:
Hello,

I'm trying to find out what the problem is and where it is located.
I have a 3 nodes Cassandra cluster (1.2.1), RF=3.
I have a keyspace and a cf as defined (using PropertyFileSnitch):

CREATE KEYSPACE backend WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'DC-TORONTO': '3'
};

USE backend;

CREATE TABLE avatars (
  id bigint PRIMARY KEY,
  avatar blob,
  image_type text
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

Status of the cluster:
Datacenter: DC-TORONTO
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns   Host ID                               Rack
UN  10.11.1.109       44.98 MB   256     46.8% 726689df-edc3-49a0-b680-370953994a8c  RAC2
UN  10.11.1.200       6.57 MB    64      10.3% d6d700d4-28aa-4722-b215-a6a7d304b8e7  RAC3
UN  10.11.1.108       54.32 MB   256     42.8% 73cd86a9-4efb-4407-9fe8-9a1b3a277af7  RAC1

I'm trying to read my writes, by using CQL (datastax-java-driver), using LOCAL_QUORUM for reads and writes. For some reason, some of the writes are lost. Not sure if it is a driver issue or cassandra issue.
Dinging further, using cqlsh client (1.2.1), I found a strange situation:

select count(*) from avatars;

 count
-------
   226

select id from avatars;

 id
---------
     314
     396
      19
 .........    ->  77 rows in result

select id, image_type from avatars;

 id      | image_type
---------+------------
     332 |        png
     314 |        png
     396 |       jpeg
      19 |        png
 1250014 |       jpeg
........ -> 226 rows in result.

I do not understand why for second select I'm able to retrieve just a part of the rows and not all rows.

Not sure if this is related or not to the initial problem.

Any help is really appreciated.
Thanks,
Gabi