Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of jbellis@gmail.com designates
 209.85.161.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=ZLp9FGDxf9vbfybXv1b4vKKLY4udHJWx45QVSaCHxLYBesVN5oliNI/YqewIVOMYOm
         1npd5qOGXRf205LjqAwyLph/kcU725i0fOn9xKgge1uCC5zioFnBpMxhfP2gtbgUOPYu
         dr9rlXKMPqXa8FhP6cMG3P0TVXiay6jQLVLNk=
MIME-Version: 1.0
In-Reply-To: <AANLkTinRp6j7v7gXCh24KMNFpS9Bzrw5i_a6KeG-nhhw@mail.gmail.com>
References: <AANLkTinxXbAvLOmS1EgKQcDwAGqZiIUkHKcCzlPplsH1@mail.gmail.com>
	<AANLkTikipJ3KTip-KXXvL_WhKHADPkCkNp_T13x3tYj0@mail.gmail.com>
	<AANLkTinRp6j7v7gXCh24KMNFpS9Bzrw5i_a6KeG-nhhw@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Fri, 11 Jun 2010 10:50:30 -0700
Message-ID: <AANLkTimI4H6lfGQbxjoM_vy_VS0Ol2_n5alkhX2rbaC7@mail.gmail.com>
Subject: Re: read operation is slow
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

you need to look at cfstats to see what the latency is internal to
cassandra, vs what your client is introducing

then you should probably read the comments in the configuration file
about caching

On Fri, Jun 11, 2010 at 9:38 AM, Caribbean410 <caribbean410@gmail.com> wrot=
e:
>
> Thanks Riyad.
>
> Right now I am just testing Cassandra on single node. The server and clie=
nt
> are running on the same machine. I tried the read test again on two
> machines, on one machine the cpu usage is around 30% most of the time and
> another is 90%.
>
> Pelops is one way to access Cassandra, there are also other java client l=
ike
> hector and jassandra, will these java clients have significant different
> performance?
>
> Also I once tried to change the storage configure file, like change
> CommitLogDirectory and DataFileDirectory to different disks, change
> DiskAccessMode to mmap for a 64bit machine, and change ConcurrentReads fr=
om
> 8 to 2. All of these do not change performance much.
>
> For other users who use different access client, like using php, c++,
> python, etc, if you have any experience in boosting the read performance,
> you are more than welcome to share with me. Thanks,
>
> On Fri, Jun 11, 2010 at 8:19 AM, Riyad Kalla <rkalla@gmail.com> wrote:
>>
>> Caribbean410,
>>
>> This comes up on the Redis list alot as well -- what you are actually
>> measuring is the client sending a network connection to the Cas server a=
nd
>> it replying -- so the performance numbers you are getting can easily be =
70%
>> network wait time and not necessarily hardcore read/write server
>> performance.
>> One way to see if this is the case, run your read test, then watch the C=
PU
>> on the server for the Cassandra process and see if it's pegging the CPU =
--
>> if it's just sitting there banging between 0-10%, the you are spending m=
ost
>> of your time waiting on network i/o (open/close sockets, etc.)
>> If you can parallelize your test to spawn say 5 threads that all do the
>> same thing, see if the performance for each thread increases=A0linearly =
--
>> which would indicate Cassandra is plenty fast in your setup, you just ne=
ed
>> to utilize more client threads over the network.
>> That new Java library, Pelops by Dominic
>> (http://ria101.wordpress.com/2010/06/11/pelops-the-beautiful-cassandra-d=
atabase-client-for-java/)
>> has a nice intrinsic node-balancing design that could be handy IF you ar=
e
>> using multiple nodes. If you are just testing against 1 node, then spawn
>> multiple threads of your code above and see how each thread's performanc=
e
>> scales.
>> -R
>> On Thu, Jun 10, 2010 at 2:39 PM, Caribbean410 <caribbean410@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I am testing the performance of cassandra. We write 200k records to
>>> database and each record is 1k size. Then we read these 200k records.
>>> It takes more than 400s to finish the read which is much slower than
>>> mysql (20s around). I read some discussion online and someone suggest
>>> to make multiple connections to make it faster. But I am not sure how
>>> to do it, do I need to change my storage setting file or just change
>>> the java client code?
>>>
>>> Here is my read code,
>>>
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Properties info =3D new Propert=
ies();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 info.put(DriverManager.CONSISTE=
NCY_LEVEL,
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 Consistency=
Level.ONE.toString());
>>>
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 IConnection connection =3D Driv=
erManager.getConnection(
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 "thrift=
://localhost:9160", info);
>>>
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 // 2. Get a KeySpace by nam=
e
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 IKeySpace keySpace =3D
>>> connection.getKeySpace("Keyspace1");
>>>
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 // 3. Get a ColumnFamily by=
 name
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 IColumnFamily cf =3D
>>> keySpace.getColumnFamily("Standard2");
>>>
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ByteArray nameFirst =3D Byt=
eArray.ofASCII("first");
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ICriteria criteria =3D cf.c=
reateCriteria();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 long readBytes =3D 0;
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 long start =3D System.curre=
ntTimeMillis();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 for (int i =3D 0; i=
 < numOfRecords; i++) {
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 int=
 n =3D random.nextInt(numOfRecords);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 userName =3D keySet[n];
>>>
>>> criteria.keyList(Lists.newArrayList(userName)).columnRange(nameFirst,
>>> nameFirst, 10);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 Map<String, List<IColumn>> map =3D
>>> criteria.select();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 List<IColumn> list =3D
>>> map.get(userName);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 ByteArray bloc =3D
>>> list.get(0).getValue();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 byte[] byteArrayloc =3D
>>> bloc.toByteArray();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 loc =3D new String(byteArrayloc);
>>> //=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 System.out.println(userName+"
>>> "+loc);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 readBytes =3D readBytes +
>>> loc.length();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 }
>>>
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 long finish=3DSystem.cu=
rrentTimeMillis();
>>>
>>> I once commented these lines
>>>
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 ByteArray bloc =3D
>>> list.get(0).getValue();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 byte[] byteArrayloc =3D
>>> bloc.toByteArray();
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 loc =3D new String(byteArrayloc);
>>> //=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 System.out.println(userName+"
>>> "+loc);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 readBytes =3D readBytes +
>>> loc.length();
>>>
>>> And the performance doesn't improve much.
>>>
>>> Any suggestion is welcome. Thanks,
>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com