incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Morton <aa...@thelastpickle.com>
Subject Re: Help! Cassandra Data Loader threads are getting stuck
Date Mon, 26 Jul 2010 21:31:32 GMT
Try running it without threading to see if it's a cassandra problem or an issue with your threading.

Perhaps split the file and run many single threaded processes to load the data.

Aaron


On 27 Jul, 2010,at 07:14 AM, Rana Aich <aichrana@gmail.com> wrote:

> Hi All,
>
> I have to load huge quantity of data into Cassandra (~10Billion rows). 
>
> I'm trying to load the Data from files using multithreading.
>
> The idea is each thread will read the TAB delimited file and process chunk of records.
>
> For example Thread1 reads line 1-1000 lines
> Thread 2 reads line 1001-2000 and insert into Cassandra.
> Thread 3 reads line 2001-3000 and insert into Cassandra.
>
> Thread 10 reads line 9001-10000 and insert into Cassandra.
> Thread 1  reads line 10001-11000 and insert into Cassandra.
> Thread 2 reads line 11001-12000 and insert into Cassandra.
>
> and so on...
>
> I'm testing with a small file size with 200000 records.
>
> But somehow the process gets stuck and doesn't proceed any further after processing say
16,000 records.
>
> I've attached my working file.
>
> Any help will be very much appreciated.
>
> Regards
>
> raich

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
    • Unnamed multipart/related (inline, None, 0 bytes)
View raw message