Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: <CAA0zji4K8=8iLHmTzpvwRy0CAZVuTNGZyxGNRy3sv4NJNk=TOw@mail.gmail.com>
References: <CAA0zji4K8=8iLHmTzpvwRy0CAZVuTNGZyxGNRy3sv4NJNk=TOw@mail.gmail.com>
From: DuyHai Doan <doanduyhai@gmail.com>
Date: Mon, 3 Oct 2016 20:10:56 +0200
Message-ID: <CABNXB2Asv4aocR1=sszaCft+GXU9dfR6EZqvtc7aBJofLhxkNw@mail.gmail.com>
Subject: Re: An extremely fast cassandra table full scan utility
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a1140f5b4b991bf053df9de74
archived-at: Mon, 03 Oct 2016 18:11:24 -0000

--001a1140f5b4b991bf053df9de74
Content-Type: text/plain; charset=UTF-8

Hello Siddarth

I just throw an eye over the architecture diagram. The idea of using
multiple threads, one for each token range is great. It help maxing out
parallelism.

With https://issues.apache.org/jira/browse/CASSANDRA-11521 it would be even
faster.

On Mon, Oct 3, 2016 at 7:51 PM, siddharth verma <sidd.verma29.list@gmail.com
> wrote:

> Hi,
> I was working on a utility which can be used for cassandra full table
> scan, at a tremendously high velocity, cassandra fast full table scan.
> How fast?
> The script dumped ~ 229 million rows in 116 seconds, with a cluster of
> size 6 nodes.
> Data transfer rates were upto 25MBps was observed on cassandra nodes.
>
> For some use case, a spark cluster was required, but for some reason we
> couldn't create spark cluster. Hence, one may use this utility to iterate
> through the entire table at very high speed.
>
> But now for any full scan, I use it freely for my adhoc java programs to
> manipulate or aggregate cassandra data.
>
> You can customize the options, setting fetch size, consistency level,
> degree of parallelism(number of threads) according to your need.
>
> You can visit https://github.com/siddv29/cfs to go through the code, see
> the logic behind it, or try it in your program.
> A sample program is also provided.
>
> I coded this utility in java.
>
> Bhuvan Rawal(bhu1rawal@gmail.com) and I worked on this concept.
> For python you may visit his blog(http://casualreflections.
> io/tech/cassandra/python/Multiprocess-Producer-Cassandra-Python) and
> github(https://gist.github.com/bhuvanrawal/93c5ae6cdd020de47e0981d36d2c07
> 85)
>
> Looking forward to your suggestions and comments.
>
> P.S. Give it a try. Trust me, the iteration speed is awesome!!
> It is a bare application, built asap. If you would like to contribute to
> the java utility, add or build up on it, do reach out
> sidd.verma29.lists@gmail.com
>
> Thanks and Regards,
> Siddharth Verma
> (previous email id on this mailing list : verma.siddharth@snapdeal.com)
>

--001a1140f5b4b991bf053df9de74
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hello Siddarth<div><br></div><div>I just throw an eye over=
 the architecture diagram. The idea of using multiple threads, one for each=
 token range is great. It help maxing out parallelism.</div><div><br></div>=
<div>With=C2=A0<a href=3D"https://issues.apache.org/jira/browse/CASSANDRA-1=
1521">https://issues.apache.org/jira/browse/CASSANDRA-11521</a> it would be=
 even faster.</div></div><div class=3D"gmail_extra"><br><div class=3D"gmail=
_quote">On Mon, Oct 3, 2016 at 7:51 PM, siddharth verma <span dir=3D"ltr">&=
lt;<a href=3D"mailto:sidd.verma29.list@gmail.com" target=3D"_blank">sidd.ve=
rma29.list@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex=
"><div dir=3D"ltr">Hi,<div>I was working on a utility which can be used for=
 cassandra full table scan, at a tremendously high velocity, cassandra fast=
 full table scan.</div><div>How fast?</div><div>The script dumped ~ 229 mil=
lion rows<span style=3D"white-space:pre-wrap">=C2=A0in </span>116 seconds, =
with a cluster of size 6 nodes.<br></div><div>Data transfer rates were upto=
 25MBps was observed on cassandra nodes.<br></div><div><br></div><div>For s=
ome use case, a spark cluster was required, but for some reason we couldn&#=
39;t create spark cluster. Hence, one may use this utility to iterate throu=
gh the entire table at very high speed.</div><div><br></div><div>But now fo=
r any full scan, I use it freely for my adhoc java programs to manipulate o=
r aggregate cassandra data.</div><div><br></div><div>You can customize the =
options, setting fetch size, consistency level, degree of parallelism(numbe=
r of threads) according to your need.</div><div><br></div><div>You can visi=
t=C2=A0<a href=3D"https://github.com/siddv29/cfs" target=3D"_blank">https:/=
/github.com/<wbr>siddv29/cfs</a> to go through the code, see the logic behi=
nd it, or try it in your program.</div><div>A sample program is also provid=
ed.</div><div><br></div><div>I coded this utility in java.</div><div><br></=
div><div>Bhuvan Rawal(<a href=3D"mailto:bhu1rawal@gmail.com" target=3D"_bla=
nk">bhu1rawal@gmail.com</a>) and I worked on this concept.=C2=A0</div><div>=
For python you may visit his blog(<a href=3D"http://casualreflections.io/te=
ch/cassandra/python/Multiprocess-Producer-Cassandra-Python" target=3D"_blan=
k">http://casualreflections.<wbr>io/tech/cassandra/python/<wbr>Multiprocess=
-Producer-<wbr>Cassandra-Python</a>) and github(<a href=3D"https://gist.git=
hub.com/bhuvanrawal/93c5ae6cdd020de47e0981d36d2c0785" target=3D"_blank">htt=
ps://gist.github.<wbr>com/bhuvanrawal/<wbr>93c5ae6cdd020de47e0981d36d2c07<w=
br>85</a>)</div><div><br></div><div>Looking forward to your suggestions and=
 comments.</div><div><br></div><div>P.S. Give it a try. Trust me, the itera=
tion speed is awesome!!</div><div>It is a bare application, built asap. If =
you would like to contribute to the java utility, add or build up on it, do=
 reach out <a href=3D"mailto:sidd.verma29.lists@gmail.com" target=3D"_blank=
">sidd.verma29.lists@gmail.com</a></div><div><br></div><div>Thanks and Rega=
rds,</div><div>Siddharth Verma</div><div>(previous email id on this mailing=
 list : <a href=3D"mailto:verma.siddharth@snapdeal.com" target=3D"_blank">v=
erma.siddharth@snapdeal.com</a>)</div></div>
</blockquote></div><br></div>

--001a1140f5b4b991bf053df9de74--