incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sylvain Lebresne <sylv...@datastax.com>
Subject Re: Trying to identify the problem with CQL ...
Date Tue, 05 Mar 2013 17:41:16 GMT
You do seem to be experiencing a problem indeed, but hard to say what it is
from just this (could be anything from a configuration problem, a cassandra
bug or a bug in the java driver).
But since you seem to be able to reproduce easily, if you can provide a
script that reproduce that issue, I'd gladly have a look (feel free to send
it to me privately if you prefer).

--
Sylvain


On Tue, Mar 5, 2013 at 4:40 PM, Gabriel Ciuloaica <gciuloaica@gmail.com>wrote:

>  So, I have added more logging to the test app (comments inline). For some
> reason I'm loosing updates.  In a for loop I'm executing upload, read
> writetime, download blob. Executed 10 times... See iteration number 2 and
> 3....
>
> 1. initialize session
>
> 0    [main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /
> 10.11.1.109 added
> 1    [main] INFO  com.datastax.driver.core.Cluster  - New Cassandra host /
> 10.11.1.200 added
>
>
>
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:214 bytes      --> uploading a blob (INSERT)
> UPLOAD:1154 ms
> Write time: 1362497519584000 ms --> reading writetime(avatar)
> DOWNLOAD LENGTH: 214 bytes   --> download the blob (SELECT)
> DOWNLOAD: 134 ms
> ---> md5 of the blob on
> upload                                                             ---> md5
> of the blob on download
> Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 ===  Download Digest
> MD5 : b1944c41a25192520d33d15d00db2718
>  -------------------------------- 0
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:4031 bytes
> UPLOAD:675 ms
> Write time: 1362497521493000 ms
> DOWNLOAD LENGTH: 4031 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 ===  Download Digest
> MD5 : 20b71b77f90b3f8ae8995a7ce7f68295
>  -------------------------------- 1
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:3392 bytes
> UPLOAD:668 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 ===  Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
>  -------------------------------- 2
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:253 bytes
> UPLOAD:668 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 ===  Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
>  -------------------------------- 3
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:266 bytes
> UPLOAD:704 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 ===  Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
>  -------------------------------- 4
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:3082 bytes
> UPLOAD:901 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 ===  Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
>  -------------------------------- 5
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:1481 bytes
> UPLOAD:703 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf ===  Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
>  -------------------------------- 6
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:5214 bytes
> UPLOAD:801 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 134 ms
> Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 ===  Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
>  -------------------------------- 7
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:2992 bytes
> UPLOAD:665 ms
> Write time: 1362497567779000 ms
> DOWNLOAD LENGTH: 2992 bytes
> DOWNLOAD: 134 ms
> Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 ===  Download Digest
> MD5 : 0848513c1b4214adf73c6ea5509ec294
>  -------------------------------- 8
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:3670 bytes
> UPLOAD:672 ms
> Write time: 1362497567779000 ms
> DOWNLOAD LENGTH: 2992 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b ===  Download Digest
> MD5 : 0848513c1b4214adf73c6ea5509ec294
>  -------------------------------- 9
>
>
> Thanks,
> Gabi
>
> On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote:
>
> Hi Sylvain,
>
> thanks for fast answer. I have updated keyspace definition and
> cassandra-topologies.properties to all 3 nodes and restarted each node.
> Both problems are still reproducible. I'm not able to read my writes and
> also the selects shows same data as in my previous email.
>
> for write and read I'm using:
> private static final String WRITE_STATEMENT = "INSERT INTO avatars (id,
> image_type, avatar) VALUES (?,?,?);";
> private static final String READ_STATEMENT = "SELECT avatar, image_type
> FROM avatars WHERE id=?";
>
> I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls.
>
> Write snippet:
>
> Session session;
>         try {
>             session = cassandraSession.getSession();
>             BoundStatement stmt = session.prepare(WRITE_STATEMENT)
>
> .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
>             stmt.enableTracing();
>             stmt.setLong("id", accountId);
>             stmt.setString("image_type", image.getType());
>             stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes()));
>             ResultSet result = session.execute(stmt);
>             LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace()
>                     .getCoordinator().getCanonicalHostName());
>
>         } catch (NoHostAvailableException e) {
>             LOG.error("Could not prepare the statement.", e);
>             throw new StorageUnavailableException(e);
>         } finally {
>             cassandraSession.releaseSession();
>         }
>
> Read snippet:
>
> Session session = null;
>         byte[] imageBytes = null;
>         String imageType = "png";
>         try {
>             session = cassandraSession.getSession();
>             BoundStatement stmt = session.prepare(READ_STATEMENT)
>
> .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
>             stmt.setLong("id", accountId);
>             ResultSet result = session.execute(stmt);
>             Iterator<Row> it = result.iterator();
>             ByteBuffer avatar = null;
>
>             while (it.hasNext()) {
>                 Row row = it.next();
>                 avatar = row.getBytes("avatar");
>                 imageType = row.getString("image_type");
>             }
>             if (avatar == null) {
>                 throw new AvatarNotFoundException("Avatar hasn't been
> found");
>             }
>             int length = avatar.remaining();
>             imageBytes = new byte[length];
>             avatar.get(imageBytes, 0, length);
>
>         } catch (NoHostAvailableException e) {
>             LOG.error("Could not prepare the statement.", e);
>             throw new StorageUnavailableException(e);
>         } finally {
>             cassandraSession.releaseSession();
>         }
>
> Let me know what other information is need it.
>
> Thanks,
> Gabi
>
>
> On 3/5/13 12:52 PM, Sylvain Lebresne wrote:
>
> Without looking into details too closely, I'd say you're probably hitting
> https://issues.apache.org/jira/browse/CASSANDRA-5292 (since you use
> NTS+propertyFileSnitch+a DC name in caps).
>
>  Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as
> dc-toronto, which then probably don't match what you have in you property
> file. This will be fixed in 1.2.3. In the meantime, a workaround would be
> to use the cassandra-cli to create/update your keyspace definition.
>
>  --
> Sylvain
>
>
> On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica <gciuloaica@gmail.com>wrote:
>
>> Hello,
>>
>> I'm trying to find out what the problem is and where it is located.
>> I have a 3 nodes Cassandra cluster (1.2.1), RF=3.
>> I have a keyspace and a cf as defined (using PropertyFileSnitch):
>>
>> CREATE KEYSPACE backend WITH replication = {
>>   'class': 'NetworkTopologyStrategy',
>>   'DC-TORONTO': '3'
>> };
>>
>> USE backend;
>>
>> CREATE TABLE avatars (
>>   id bigint PRIMARY KEY,
>>   avatar blob,
>>   image_type text
>> ) WITH
>>   bloom_filter_fp_chance=0.010000 AND
>>   caching='KEYS_ONLY' AND
>>   comment='' AND
>>   dclocal_read_repair_chance=0.000000 AND
>>   gc_grace_seconds=864000 AND
>>   read_repair_chance=0.100000 AND
>>   replicate_on_write='true' AND
>>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>>   compression={'sstable_compression': 'SnappyCompressor'};
>>
>> Status of the cluster:
>> Datacenter: DC-TORONTO
>> ======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns   Host ID
>>             Rack
>> UN  10.11.1.109       44.98 MB   256     46.8%
>> 726689df-edc3-49a0-b680-370953994a8c  RAC2
>> UN  10.11.1.200       6.57 MB    64      10.3%
>> d6d700d4-28aa-4722-b215-a6a7d304b8e7  RAC3
>> UN  10.11.1.108       54.32 MB   256     42.8%
>> 73cd86a9-4efb-4407-9fe8-9a1b3a277af7  RAC1
>>
>> I'm trying to read my writes, by using CQL (datastax-java-driver), using
>> LOCAL_QUORUM for reads and writes. For some reason, some of the writes are
>> lost. Not sure if it is a driver issue or cassandra issue.
>> Dinging further, using cqlsh client (1.2.1), I found a strange situation:
>>
>> select count(*) from avatars;
>>
>>  count
>> -------
>>    226
>>
>> select id from avatars;
>>
>>  id
>> ---------
>>      314
>>      396
>>       19
>>  .........    ->  77 rows in result
>>
>> select id, image_type from avatars;
>>
>>  id      | image_type
>> ---------+------------
>>      332 |        png
>>      314 |        png
>>      396 |       jpeg
>>       19 |        png
>>  1250014 |       jpeg
>> ........ -> 226 rows in result.
>>
>> I do not understand why for second select I'm able to retrieve just a
>> part of the rows and not all rows.
>>
>> Not sure if this is related or not to the initial problem.
>>
>> Any help is really appreciated.
>> Thanks,
>> Gabi
>>
>>
>>
>>
>>
>
>
>

Mime
View raw message