Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Date: Thu, 17 Nov 2016 13:58:00 +0000 (UTC)
From: Joe Olson <technology@nododos.com>
To: user@cassandra.apache.org
Message-ID: <1616103142.1405.1479391080488.JavaMail.zimbra@nododos.com>
Subject: Any Bulk Load on Large Data Set Advice?
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_1404_1294362120.1479391080479"
Thread-Topic: Any Bulk Load on Large Data Set Advice?
Thread-Index: KvOC6lKui4WX4Y4Q/3cWpyycrL72RQ==
archived-at: Thu, 17 Nov 2016 14:05:07 -0000

------=_Part_1404_1294362120.1479391080479
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

I received a grant to do some analysis on netflow data (Local IP address, Local Port, Remote IP address, Remote Port, time, # of packets, etc) using Cassandra and Spark. The de-normalized data set is about 13TB out the door. I plan on using 9 Cassandra nodes (replication factor=3) to store the data, with Spark doing the aggregation. 

Data set will be immutable once loaded, and am using the replication factor = 3 to somewhat simulate the real world. Most of the analysis will be of the sort "Give me all the remote ip addresses for source IP 'X' between time t1 and t2" 

I built and tested a bulk loader following this example in GitHub: https://github.com/yukim/cassandra-bulkload-example to generate the SSTables, but I have not executed it on the entire data set yet. 

Any advice on how to execute the bulk load under this configuration? Can I generate the SSTables in parallel? Once generated, can I write the SSTables to all nodes simultaneously? Should I be doing any kind of sorting by the partition key? 

This is a lot of data, so I figured I'd ask before I pulled the trigger. Thanks in advance! 


------=_Part_1404_1294362120.1479391080479
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s=
ize: 12pt; color: #000000"><div>I received a grant to do some analysis on n=
etflow data (Local IP address, Local Port, Remote IP address, Remote Port, =
time, # of packets, etc) using Cassandra and Spark. The de-normalized data =
set is about 13TB out the door. I plan on using 9 Cassandra nodes (replicat=
ion factor=3D3) to store the data, with Spark doing the aggregation. <br da=
ta-mce-bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><div><div>Data=
 set will be immutable once loaded, and am using the replication factor =3D=
 3 to somewhat simulate the real world. Most of the analysis will be of the=
 sort "Give me all the remote ip addresses for source IP 'X' between time t=
1 and t2"<br></div></div><div><br data-mce-bogus=3D"1"></div><div>I built a=
nd tested a bulk loader following this example in GitHub: https://github.co=
m/yukim/cassandra-bulkload-example to generate the SSTables, but I have not=
 executed it on the entire data set yet.</div><div><br data-mce-bogus=3D"1"=
></div><div>Any advice on how to execute the bulk load under this configura=
tion?&nbsp; Can I generate the SSTables in parallel? Once generated, can I =
write the SSTables to all nodes simultaneously? Should I be doing any kind =
of sorting by the partition key?<br></div><div><br data-mce-bogus=3D"1"></d=
iv><div>This is a lot of data, so I figured I'd ask before I pulled the tri=
gger. Thanks in advance!<br data-mce-bogus=3D"1"></div><div><br data-mce-bo=
gus=3D"1"></div><div><br data-mce-bogus=3D"1"></div></div></body></html>
------=_Part_1404_1294362120.1479391080479--