cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Malcolm Smith <malsm...@treehousesystems.com>
Subject Re: Help! Cassandra Data Loader threads are getting stuck
Date Mon, 26 Jul 2010 21:40:04 GMT
Also make sure you have consistency level set to at least ONE

Sent from my iPhone

On Jul 26, 2010, at 5:31 PM, Aaron Morton <aaron@thelastpickle.com> wrote:

> Try running it without threading to see if it's a cassandra problem or an issue with
your threading. 
> 
> Perhaps split the file and run many single threaded processes to load the data. 
> 
> Aaron
> 
> 
> On 27 Jul, 2010,at 07:14 AM, Rana Aich <aichrana@gmail.com> wrote:
> 
>> Hi All,
>> 
>> I have to load huge quantity of data into Cassandra (~10Billion rows). 
>> 
>> I'm trying to load the Data from files using multithreading.
>> 
>> The idea is each thread will read the TAB delimited file and process chunk of records.
>> 
>> For example Thread1 reads line 1-1000 lines
>> Thread 2 reads line 1001-2000 and insert into Cassandra.
>> Thread 3 reads line 2001-3000 and insert into Cassandra.
>> 
>> Thread 10 reads line 9001-10000 and insert into Cassandra.
>> Thread 1  reads line 10001-11000 and insert into Cassandra.
>> Thread 2 reads line 11001-12000 and insert into Cassandra.
>> 
>> and so on...
>> 
>> I'm testing with a small file size with 200000 records.
>> 
>> But somehow the process gets stuck and doesn't proceed any further after processing
say 16,000 records.
>> 
>> I've attached my working file.
>> 
>> Any help will be very much appreciated.
>> 
>> Regards
>> 
>> raich

Mime
View raw message