Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates
 209.85.214.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABxBLH8X0XmLycAaDoJ_qaE6O1hTWFpip_brq2cmrGGusDJPPA@mail.gmail.com>
References: 
 <CABxBLH_NAZ9M1p=Hav36xQ+ZHMSbOSy1-JJjkxYzJQp1HQkg6A@mail.gmail.com>
 <CAAam9ssrHaPXFpXN_TpNKoezM9KZ39Zf0H43N3G5Tn8fLoOyUg@mail.gmail.com>
 <CABxBLH80rMZUwB-KN5=teg6jDx0ifvC=NwcY3c7-R90+8wi8NQ@mail.gmail.com>
 <8FADD602-74E1-475C-A323-6FB68F41B7AB@thelastpickle.com>
 <CABxBLH8jkJsCOEtjnhddStrTH2RZks7bk+jmBouanBrVFVYZoA@mail.gmail.com>
 <64AC8E12-7E5B-425E-9414-EE6BC6A4670D@thelastpickle.com>
 <CABxBLH8X0XmLycAaDoJ_qaE6O1hTWFpip_brq2cmrGGusDJPPA@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Sat, 19 May 2012 12:07:22 -0500
Message-ID: 
 <CALdd-zhq534C-G3SyYy3HRUAaOYDTJFzQqoSjDjDPnMim=iBQw@mail.gmail.com>
Subject: Re: need some clarification on recommended memory size
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

So, you're doing about 20 ops/s where each op consists of "read 2
metadata columns, then read ~250 columns of ~2K each."  Is that right?

Is your test client multithreaded?  Is it on a separate machine from
the Cassandra server?

What is your bottleneck?
http://spyced.blogspot.com/2010/01/linux-performance-basics.html

On Thu, May 17, 2012 at 1:08 PM, Yiming Sun <yiming.sun@gmail.com> wrote:
> Hi Aaron,
>
> Thank you for guiding us by breaking down the issue. =A0Please see my ans=
wers
> embedded
>
>> Is this a single client ?
>
> Yes
>
>> How many columns is it asking for ?
>
> the client knows a list of all row keys, and it randomly picks 100, and
> loops 100 times. =A0It first reads a metadata column to figure out how ma=
ny
> columns to read, and it then reads these columns
>
>> What sort of query are you sending, slice or named columns?
>
> currently all queries are slice queries. =A0so the first slice query read=
s the
> metadata column (actually 2 metadata columns, one is for Number of column=
s
> to read, the other for other information which is not needed for the purp=
ose
> of performance test, but I kept it in there to make it similar to the rea=
l
> situation). =A0 =A0It then generates the column name array and sends the =
second
> slice query.
>
> The timing for the queries is completely isolated, and excludes the time
> spent generating column name array etc.
>
>
>> =A0From the client side how long is a single read taking ?
>
> I am not 100% sure on what you are asking... are you saying how long it
> takes for SliceQuery.execute()? =A0The average we are getting are between
> 50-70 ms, and nodetool report similar latency, differ by 5-10ms at top.
>
>
>> What is the write workload like? =A0it sounds like it's write once read
>> many.
>
> Indeed it is like a WORM environment. For the performance, we don't have =
any
> writes.
>
>>=A0memory speed > network speed
>
> yes. =A0right now, our data is only a sample about 250K rows, so the defa=
ult
> 200,000 key cache hits above 90%. =A0But we soon will be hosting the real=
 deal
> with about 3M rows, so I am not sure our memory size will be able to keep=
 up
> with it.
>
> In any case, Aaron, please let us know if you have any
> suggestions/comments/insights. =A0Thanks!
>
> -- Y.
>
>
> On Thu, May 17, 2012 at 1:04 AM, aaron morton <aaron@thelastpickle.com>
> wrote:
>>
>> The read rate that I have been seeing is about 3MB/sec, and that is
>> reading the raw bytes... using string serializer the rate is even lower,
>> about 2.2MB/sec.
>>
>> Can we break this down a bit:
>>
>> Is this a single client ?
>> How many columns is it asking for ?
>> What sort of query are you sending, slice or named columns?
>> From the client side how long is a single read taking ?
>> What is the write workload like? =A0it sounds like it's write once read
>> many.
>>
>> Use nodetool cfstats to see what the read latency is on a single node.
>> (see=A0http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/) Is th=
ere
>> much difference between this and the latency from the client perspective=
 ?
>>
>>
>>
>> Using JNA may help, but a blog article seems to say it only increase 13%=
,
>> which is not very significant when the base performance is in single-dig=
it
>> MBs.
>>
>> There are other reasons to have JNA installed: more efficient snapshots
>> and advising the OS when file operations should not be cached.
>>
>> =A0Our environment is virtualized, and the disks are actually SAN throug=
h
>> fiber channels, so I don't know if that has impact on performance as wel=
l.
>>
>> memory speed > network speed
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com