Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: softfail (athena.apache.org: transitioning domain of jrf@mit.edu
 does not designate 69.164.218.4 as permitted sender)
Date: Thu, 9 May 2013 21:09:06 -0400 (EDT)
From: "John R. Frank" <jrf@mit.edu>
To: user@cassandra.apache.org
Subject: pycassa failures in large batch cycling
Message-ID: <alpine.DEB.2.00.1305092046220.13940@computableinsights.com>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII

C* users,

We have a process that loads a large batch of rows from Cassandra into 
many separate compute workers.  The rows are one-column wide and range in 
size for a couple KB to ~100 MB.  After manipulating the data for a while, 
each compute worker writes the data back with *new* row keys computed by 
the workers (UUIDs).

After the full batch is written back to new rows, a cleanup worker deletes 
the old rows.

After several cycles, pycassa starts getting connection failures.

Should we use a pycassa listener to catch these failures and just recreate 
the ConnectionPool and keep going as if the connection had not dropped? 
Or is there a better approach?

These failures happen on just a simple single-node setup with a total data 
set less than half the size of Java heap space, e.g. 2GB data (times two 
for the two copies during cycling) versus 8GB heap.  We tried reducing 
memtable_flush_queue_size to 2 so that it would flush the deletes faster, 
and also tried multithreaded_compaction=true, but still pycassa gets 
connection failures.

Is this expected before for shedding load?  Or is this unexpected?

Would things be any different if we used multiple nodes and scaled the 
data and worker count to match?  I mean, is there something inherent to 
cassandra's operating model that makes it want to always have multiple 
nodes?

Thanks for pointers,
John