incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Richter <...@tricnet.de>
Subject Re: Existing column(s) not readable
Date Mon, 10 Oct 2011 21:35:57 GMT
Hi Aaron,

normally we use hector to access cassandra, but for debugging I switched
to cassandra-cli.

Column can not be read by a simple
get CFName['rowkey']['colname'];

Response is "Value was not found"
if i query another column, everything is just fine.

Serverlog for unsuccessful read (keyspace and CF names replaced):

DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,739 CassandraServer.java
(line 280) get

DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,744 StorageProxy.java (line
320) Command/ConsistencyLevel is
SliceByNamesReadCommand(table='Keyspace',
key=61636162626139322d396638312d343562382d396637352d393162303337383030393762,
columnParent='QueryPath(columnFamilyName='ColumnFamily',
superColumnName='null', columnName='null')',
columns=[574c303030375030,])/ONE

DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,750 ReadCallback.java (line
86) Blockfor/repair is 1/true; setting up requests to localhost/127.0.0.1

DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,750 StorageProxy.java (line
343) reading data locally

DEBUG [ReadStage:33] 2011-10-10 23:15:29,751 StorageProxy.java (line
448) LocalReadRunnable reading SliceByNamesReadCommand(table='Keyspace',
key=61636162626139322d396638312d343562382d396637352d393162303337383030393762,
columnParent='QueryPath(columnFamilyName='ColumnFamily',
superColumnName='null', columnName='null')', columns=[574c303030375030,])

DEBUG [pool-1-thread-1] 2011-10-10 23:15:29,818 StorageProxy.java (line
393) Read: 67 ms.

Log looks fine to me, but no result is returned.

Best,

Thomas

On 10/10/2011 10:00 PM, aaron morton wrote:
> How are they unreadable ? You need to go into some details about what is going wrong.

> 
> What sort of read ? 
> What client ? 
> What is in the logging on client and server side ? 
> 
> 
> Try turning the logging up to DEBUG on the server to watch what happens. 
> 
> Cheers
>  
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 10/10/2011, at 9:23 PM, Thomas Richter wrote:
> 
>> Hi,
>>
>> no errors in the server logs. The columns are unreadable on all nodes at
>> any consistency level (ONE, QUORUM, ALL). We started with 0.7.3 and
>> upgraded to 0.7.6-2 two months ago.
>>
>> Best,
>>
>> Thomas
>>
>> On 10/10/2011 10:03 AM, aaron morton wrote:
>>> What error are you seeing  in the server logs ? Are the columns unreadable at
all Consistency Levels ? i.e. are the columns unreadable on all nodes.
>>>
>>> What is the upgrade history of the cluster ? What version did it start at ? 
>>>
>>> Cheers
>>>
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Cassandra Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 10/10/2011, at 7:42 AM, Thomas Richter wrote:
>>>
>>>> Hi,
>>>>
>>>> here is some further information. Compaction did not help, but data is
>>>> still there when I dump the row with sstable2json.
>>>>
>>>> Best,
>>>>
>>>> Thomas
>>>>
>>>> On 10/08/2011 11:30 PM, Thomas Richter wrote:
>>>>> Hi,
>>>>>
>>>>> we are running a 3 node cassandra (0.7.6-2) cluster and some of our
>>>>> column families contain quite large rows (400k+ columns, 4-6GB row size).
>>>>> Replicaton factor is 3 for all keyspaces. The cluster is running fine
>>>>> for several months now and we never experienced any serious trouble.
>>>>>
>>>>> Some days ago we noticed, that some previously written columns could
not
>>>>> be read. This does not always happen, and only some dozen columns out
of
>>>>> 400k are affected.
>>>>>
>>>>> After ruling out application logic as a cause I dumped the row in
>>>>> question with sstable2json and the columns are there (and are not marked
>>>>> for deletion).
>>>>>
>>>>> Next thing was setting up a fresh single node cluster and copying the
>>>>> column family data to that node. Columns could not be read either.
>>>>> Right now I'm running a nodetool compact for the cf to see if data could
>>>>> be read afterwards.
>>>>>
>>>>> Is there any explanation for such behavior? Are there any suggestions
>>>>> for further investigation?
>>>>>
>>>>> TIA,
>>>>>
>>>>> Thomas
>>>>
>>>
>>
> 


Mime
View raw message