Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 16036 invoked from network); 26 Jul 2010 21:41:44 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 26 Jul 2010 21:41:44 -0000 Received: (qmail 32501 invoked by uid 500); 26 Jul 2010 21:41:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 32463 invoked by uid 500); 26 Jul 2010 21:41:42 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 32455 invoked by uid 99); 26 Jul 2010 21:41:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:41:42 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=MIME_QP_LONG_LINE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.161.172] (HELO mail-gx0-f172.google.com) (209.85.161.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jul 2010 21:41:36 +0000 Received: by gxk1 with SMTP id 1so1236650gxk.31 for ; Mon, 26 Jul 2010 14:41:15 -0700 (PDT) Received: by 10.150.149.19 with SMTP id w19mr9881464ybd.415.1280180473176; Mon, 26 Jul 2010 14:41:13 -0700 (PDT) Received: from [10.78.62.112] (mobile-166-137-139-221.mycingular.net [166.137.139.221]) by mx.google.com with ESMTPS id h8sm3998703ibk.15.2010.07.26.14.41.09 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 26 Jul 2010 14:41:10 -0700 (PDT) Subject: Re: Help! Cassandra Data Loader threads are getting stuck References: <4868d8c7-1a3b-6a43-e7dd-d43826efa7d9@me.com> From: Malcolm Smith Content-Type: text/plain; charset=us-ascii X-Mailer: iPhone Mail (8A306) In-Reply-To: <4868d8c7-1a3b-6a43-e7dd-d43826efa7d9@me.com> Message-Id: Date: Mon, 26 Jul 2010 17:40:04 -0400 To: "user@cassandra.apache.org" Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (iPhone Mail 8A306) Also make sure you have consistency level set to at least ONE Sent from my iPhone On Jul 26, 2010, at 5:31 PM, Aaron Morton wrote: > Try running it without threading to see if it's a cassandra problem or an i= ssue with your threading.=20 >=20 > Perhaps split the file and run many single threaded processes to load the d= ata.=20 >=20 > Aaron >=20 >=20 > On 27 Jul, 2010,at 07:14 AM, Rana Aich wrote: >=20 >> Hi All, >>=20 >> I have to load huge quantity of data into Cassandra (~10Billion rows).=20= >>=20 >> I'm trying to load the Data from files using multithreading. >>=20 >> The idea is each thread will read the TAB delimited file and process chun= k of records. >>=20 >> For example Thread1 reads line 1-1000 lines >> Thread 2 reads line 1001-2000 and insert into Cassandra. >> Thread 3 reads line 2001-3000 and insert into Cassandra. >>=20 >> Thread 10 reads line 9001-10000 and insert into Cassandra. >> Thread 1 reads line 10001-11000 and insert into Cassandra. >> Thread 2 reads line 11001-12000 and insert into Cassandra. >>=20 >> and so on... >>=20 >> I'm testing with a small file size with 200000 records. >>=20 >> But somehow the process gets stuck and doesn't proceed any further after p= rocessing say 16,000 records. >>=20 >> I've attached my working file. >>=20 >> Any help will be very much appreciated. >>=20 >> Regards >>=20 >> raich