Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <4E90C0E5.9050002@tricnet.de>
Date: Sat, 08 Oct 2011 23:30:13 +0200
From: Thomas Richter <tho@tricnet.de>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.2.21) Gecko/20110831 Lightning/1.0b2 Thunderbird/3.1.13
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Existing column(s) not readable
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi,

we are running a 3 node cassandra (0.7.6-2) cluster and some of our
column families contain quite large rows (400k+ columns, 4-6GB row size).
Replicaton factor is 3 for all keyspaces. The cluster is running fine
for several months now and we never experienced any serious trouble.

Some days ago we noticed, that some previously written columns could not
be read. This does not always happen, and only some dozen columns out of
400k are affected.

After ruling out application logic as a cause I dumped the row in
question with sstable2json and the columns are there (and are not marked
for deletion).

Next thing was setting up a fresh single node cluster and copying the
column family data to that node. Columns could not be read either.
Right now I'm running a nodetool compact for the cf to see if data could
be read afterwards.

Is there any explanation for such behavior? Are there any suggestions
for further investigation?

TIA,

Thomas