Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9AA8410AB0 for ; Mon, 6 May 2013 19:04:17 +0000 (UTC) Received: (qmail 72901 invoked by uid 500); 6 May 2013 19:04:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 72823 invoked by uid 500); 6 May 2013 19:04:14 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 72810 invoked by uid 99); 6 May 2013 19:04:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 May 2013 19:04:14 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of derek@fyrie.net designates 72.14.182.177 as permitted sender) Received: from [72.14.182.177] (HELO sikozu.fyrie.net) (72.14.182.177) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 May 2013 19:04:09 +0000 Received: from mail-ob0-f182.google.com ([209.85.214.182]) by sikozu.fyrie.net with esmtpsa (TLS1.0:RSA_ARCFOUR_SHA1:16) (Exim 4.76) (envelope-from ) id 1UZQhQ-0005ie-Ec for user@cassandra.apache.org; Mon, 06 May 2013 13:03:48 -0600 Received: by mail-ob0-f182.google.com with SMTP id eh20so3350088obb.41 for ; Mon, 06 May 2013 12:03:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=xXEz2Lw4iTSIdi/ZUttfogGKVLtorMigcJhxibYPpQo=; b=OQg0K4F38VBh1fhpUdJuGY/gZ+0v6GWR5cijz24GoGZgYtXPZLWtTfNlLx+daRMff4 3SR/qmRglDSJAmZyMqTEv0RPsiHmTXTaDkwvTPpLfHIfoWQOLDdR1ksMCEZehU8PrG4O D+Nz2uRztPWHKLFNUlNvxfpN+M1U9yTwlsQy/fch1SqK94IHEXNZtsIph6G6DUlLL+Xp i9GY+jzM8u4GbKznCKv6k6Y34iZZJZ35E3UAoZSQQ6k8ZaYMzMVpSjatcPuFM8PbvdRv tvksj2O1cYP38fEmGXDh0pFdg7SrfODC/pu6CtoIBWVBIt5kzkA9nz9/baQf24PWGgjb U9Mg== MIME-Version: 1.0 X-Received: by 10.60.142.103 with SMTP id rv7mr5841301oeb.34.1367867023083; Mon, 06 May 2013 12:03:43 -0700 (PDT) Received: by 10.76.86.137 with HTTP; Mon, 6 May 2013 12:03:42 -0700 (PDT) In-Reply-To: References: Date: Mon, 6 May 2013 13:03:42 -0600 Message-ID: Subject: Re: hector or astyanax From: Derek Williams To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b2e088703255004dc115cd1 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b2e088703255004dc115cd1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Also have to keep in mind that it should be rare to only use a single socket since you are usually making at least 1 connection per node in the cluster (or local datacenter). There is also nothing enforcing that a single client cannot open more than 1 connection to a node. In the end it should come down to which protocol implementation is faster. On Mon, May 6, 2013 at 11:58 AM, Aaron Turner wrote: > From my experience, your NIC buffers generally aren't the problem (or at > least it's easy to tune them to fix). It's TCP. Simply put, your raw NI= C > throughput > single TCP socket throughput on most modern hardware/OS > combinations. This is especially true as latency increases between the t= wo > hosts. This is why Bittorrent or "download accellerators" are often fast= er > then just downloading a large file via your browser or ftp client- they'r= e > running multiple TCP connections in parallel compared to only one. > > TCP is great for reliable, bi-directional, stream based communication. > Not the best solution for high throughput though. UDP is much better fo= r > that, but then you loose all the features that TCP gives you and so then > people end up re-inventing the wheel (poorly I might add). > > So yeah, I think the answer to the question of "which is faster" the > answer is "it depends on your queries". > > > > On Mon, May 6, 2013 at 10:24 AM, Hiller, Dean wrote= : > >> You have me thinking more. I wonder in practice if 3 sockets is any >> faster than 1 socket when doing nio. If your buffer sizes were small, >> maybe that would be the case. Usually the nic buffers are big so when t= he >> selector fires it is reading from 3 buffers for 3 sockets or 1 buffer fo= r >> one socket. In both cases, all 3 requests are there in the buffers. At >> any rate, my belief is it probably is still basically parallel performan= ce >> on one socket though I have not tested my theory=E2=80=A6..My theory bei= ng the real >> bottleneck on performance being the work cassandra has to do on the read= s >> and such. >> >> What about 20 sockets then(like someone has a pool). Will it be any >> faster=E2=80=A6not really sure as in the end you are still held up by th= e real >> bottleneck of reading from disk on the cassandra side. We went to 20 >> threads in one case using 20 sockets with astyanax and received no >> performance improvement(synchronous but more sockets did not improve our >> performance). Ie. It may be the case 90% of the time, one socket is jus= t >> as fast as 10/20=E2=80=A6..I would love to know the truth/answer to that= though. >> >> Later, >> Dean >> >> >> From: Aaron Turner > >> Reply-To: "user@cassandra.apache.org" = < >> user@cassandra.apache.org> >> Date: Monday, May 6, 2013 10:57 AM >> To: cassandra users > user@cassandra.apache.org>> >> Subject: Re: hector or astyanax >> >> Just because you can batch queries or have the server process them out o= f >> order doesn't make it fully "parellel". You're still using a single TCP >> connection which is by definition a serial data stream. Basically, if y= ou >> send a bunch of queries which each return a large amount of data you've >> effectively limited your query throughput to a single TCP connection. >> Using Thrift, each query result is returned in it's own TCP stream in >> *parallel*. >> >> Not saying the new API isn't great, doesn't have it's place or may have >> better performance in certain situations, but generally speaking I would >> refrain from making general claims without actual benchmarks to back the= m >> up. I do completely agree that Async interfaces have their place and h= ave >> certain advantages over multi-threading models, but it's just another to= ol >> to be used when appropriate. >> >> Just my .02. :) >> >> >> >> On Mon, May 6, 2013 at 5:08 AM, Hiller, Dean > > wrote: >> I was under the impression that it is multiple requests using a single >> connectin PARALLEL not serial as they have request ids and the responses= do >> as well so you can send a request while a previous request has no respon= se >> just yet. >> >> I think you do get a big speed advantage from the asynchronous nature as >> you do not need to hold up so many threads in your webserver while you h= ave >> outstanding requests being processed. The thrift async was not exactly >> async like I am suspecting the new java driver is, but have not verified= (I >> hope it is) >> >> Dean >> >> From: Aaron Turner > >>> >> Reply-To: "user@cassandra.apache.org> >>" < >> user@cassandra.apache.org> user@cassandra.apache.org>> >> Date: Sunday, May 5, 2013 5:27 PM >> To: cassandra users > user@cassandra.apache.org>> user@cassandra.apache.org>>> >> Subject: Re: hector or astyanax >> >> >> >> On Sun, May 5, 2013 at 1:09 PM, Derek Williams > derek@fyrie.net>>> wrote: >> The binary protocol is able to multiplex multiple requests using a singl= e >> connection, which can lead to much better performance (similar to HTTP v= s >> SPDY). This is without comparing the performance of thrift vs binary >> protocol, which I assume the binary protocol would be faster since it is >> specialized for cassandra requests. >> >> >> Curious why you think multiplexing multiple requests over a single >> connection (serial) is faster then multiple requests over multiple >> connections (parallel)? >> >> And isn't Thrift a binary protocol? >> >> >> -- >> Aaron Turner >> http://synfin.net/ Twitter: @synfinatic >> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & >> Windows >> Those who would give up essential Liberty, to purchase a little temporar= y >> Safety, deserve neither Liberty nor Safety. >> -- Benjamin Franklin >> "carpe diem quam minimum credula postero" >> >> >> >> -- >> Aaron Turner >> http://synfin.net/ Twitter: @synfinatic >> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & >> Windows >> Those who would give up essential Liberty, to purchase a little temporar= y >> Safety, deserve neither Liberty nor Safety. >> -- Benjamin Franklin >> "carpe diem quam minimum credula postero" >> > > > > -- > Aaron Turner > http://synfin.net/ Twitter: @synfinatic > http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & > Windows > Those who would give up essential Liberty, to purchase a little temporary > Safety, deserve neither Liberty nor Safety. > -- Benjamin Franklin > "carpe diem quam minimum credula postero" > --=20 Derek Williams --047d7b2e088703255004dc115cd1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Also have to keep in mind that it should be rare to only u= se a single socket since you are usually making at least 1 connection per n= ode in the cluster (or local datacenter). There is also nothing enforcing t= hat a single client cannot open more than 1 connection to a node. In the en= d it should come down to which protocol implementation is faster.


On Mon, May 6= , 2013 at 11:58 AM, Aaron Turner <synfinatic@gmail.com> w= rote:
From my experience, your NI= C buffers generally aren't the problem (or at least it's easy to tu= ne them to fix). =C2=A0It's TCP. =C2=A0Simply put, your raw NIC through= put > single TCP socket throughput on most modern hardware/OS combinatio= ns. =C2=A0This is especially true as latency increases between the two host= s. =C2=A0This is why Bittorrent or "download accellerators" are o= ften faster then just downloading a large file via your browser or ftp clie= nt- they're running multiple TCP connections in parallel compared to on= ly one.

TCP is great for reliable, bi-directional, stream based comm= unication. =C2=A0Not the best solution for high throughput though. =C2=A0UD= P is much better for that, but then you loose all the features that TCP giv= es you and so then people end up re-inventing the wheel (poorly I might add= ).

So yeah, I think the answer to the question of "wh= ich is faster" the answer is "it depends on your queries".



On Mon, May 6, 2013 at 10:24 AM, Hiller, Dea= n <Dean.Hiller@nrel.gov> wrote:
You have me thinking more. =C2=A0I wonder in practice if 3 sockets is any f= aster than 1 socket when doing nio. =C2=A0If your buffer sizes were small, = maybe that would be the case. =C2=A0Usually the nic buffers are big so when= the selector fires it is reading from 3 buffers for 3 sockets or 1 buffer = for one socket. =C2=A0In both cases, all 3 requests are there in the buffer= s. =C2=A0At any rate, my belief is it probably is still basically parallel = performance on one socket though I have not tested my theory=E2=80=A6..My t= heory being the real bottleneck on performance being the work cassandra has= to do on the reads and such.

What about 20 sockets then(like someone has a pool). =C2=A0Will it be any f= aster=E2=80=A6not really sure as in the end you are still held up by the re= al bottleneck of reading from disk on the cassandra side. =C2=A0We went to = 20 threads in one case using 20 sockets with astyanax and received no perfo= rmance improvement(synchronous but more sockets did not improve our perform= ance). =C2=A0Ie. It may be the case 90% of the time, one socket is just as = fast as 10/20=E2=80=A6..I would love to know the truth/answer to that thoug= h.

Later,
Date: Monday, May 6, 2013 10:57 AM
To: cassandra users <user@cassandra.apache.org<mailto:user@cassandra.apache.org&= gt;>
Subject: Re: hector or astyanax

Just because you can batch queries or have the server process th= em out of order doesn't make it fully "parellel". =C2=A0You&#= 39;re still using a single TCP connection which is by definition a serial d= ata stream. =C2=A0Basically, if you send a bunch of queries which each retu= rn a large amount of data you've effectively limited your query through= put to a single TCP connection. =C2=A0Using Thrift, each query result is re= turned in it's own TCP stream in *parallel*.

Not saying the new API isn't great, doesn't have it's place or = may have better performance in certain situations, but generally speaking I= would refrain from making general claims without actual benchmarks to back= them up. =C2=A0 I do completely agree that Async interfaces have their pla= ce and have certain advantages over multi-threading models, but it's ju= st another tool to be used when appropriate.

Just my .02. :)



On Mon, May 6, 2013 at 5:08 AM, Hiller, Dean <Dean.Hiller@nrel.gov<mail= to:Dean.Hiller@nr= el.gov>> wrote:
I was under the impression that it is multiple requests using a single conn= ectin PARALLEL not serial as they have request ids and the responses do as = well so you can send a request while a previous request has no response jus= t yet.

I think you do get a big speed advantage from the asynchronous nature as yo= u do not need to hold up so many threads in your webserver while you have o= utstanding requests being processed. =C2=A0The thrift async was not exactly= async like I am suspecting the new java driver is, but have not verified(I= hope it is)

Dean

From: Aaron Turner <synfinatic@gmail.com<mailto:synfinatic@gmail.com><mailto:synfinatic@gmail.com<mailto:synfi= natic@gmail.com>>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto= :user@cassan= dra.apache.org<mailto:user@cassandra.apache.org>>" <user@cassandra.apache.= org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mail= to:user@cass= andra.apache.org>>>
Date: Sunday, May 5, 2013 5:27 PM
To: cassandra users <user@cassandra.apache.org<mailto:user@cassandra.apache.org= ><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
Subject: Re: hector or astyanax



On Sun, May 5, 2013 at 1:09 PM, Derek Williams <derek@fyrie.net<mailto= :derek@fyrie.net&g= t;<mailto:derek@fyr= ie.net<mailto:d= erek@fyrie.net>>> wrote:
The binary protocol is able to multiplex multiple requests using a single c= onnection, which can lead to much better performance (similar to HTTP vs SP= DY). This is without comparing the performance of thrift vs binary protocol= , which I assume the binary protocol would be faster since it is specialize= d for cassandra requests.


Curious why you think multiplexing multiple requests over a single connecti= on (serial) is faster then multiple requests over multiple connections (par= allel)?

And isn't Thrift a binary protocol?


--
Aaron Turner
http://synfin.net/ =C2= =A0 =C2=A0 =C2=A0 =C2=A0 Twitter: @synfinatic
http://tcpreplay= .synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.
=C2=A0 =C2=A0 -- Benjamin Franklin
"carpe diem quam minimum credula postero"



--
Aaron Turner
http://synfin.net/ =C2= =A0 =C2=A0 =C2=A0 =C2=A0 Twitter: @synfinatic
http://tcpreplay= .synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.
=C2=A0 =C2=A0 -- Benjamin Franklin
"carpe diem quam minimum credula postero"



--
= Aaron Turner
http://syn= fin.net/=C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0 Twitter: @synfinatic
http://tcpreplay.synf= in.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary <= br>Safety, deserve neither Liberty nor Safety.=C2=A0
=C2=A0 =C2=A0 -- B= enjamin Franklin
"carpe diem quam minimum credula postero"



--
=
Derek Williams
--047d7b2e088703255004dc115cd1--