From user-return-30573-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Wed Dec 12 03:14:14 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D3E3BD74A for ; Wed, 12 Dec 2012 03:14:14 +0000 (UTC) Received: (qmail 48447 invoked by uid 500); 12 Dec 2012 03:14:11 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 48284 invoked by uid 500); 12 Dec 2012 03:14:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 48256 invoked by uid 99); 12 Dec 2012 03:14:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 03:14:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dongtalk@gmail.com designates 209.85.210.179 as permitted sender) Received: from [209.85.210.179] (HELO mail-ia0-f179.google.com) (209.85.210.179) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 03:14:03 +0000 Received: by mail-ia0-f179.google.com with SMTP id o25so299970iad.24 for ; Tue, 11 Dec 2012 19:13:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=wIgC+zu7alsUQ0RarQ6MoExj6bw9tFrPgchYMxXWidY=; b=ys0oB09qF3Eskok7lwGYX4Q6Wm6qGDLFUAN61+nQ341/EJr4buAzL/L9QY36+RXvJb 9d46CrJ/3YnMIY3LWry/kC03xgHh8IzgOkIiz0Tup2Uaqim8mLfWGJ9aA96JoxDrhrx0 KfWnG9QR8nEurFyvekgMLvwv3AocHCzxtZkP/cwUkRwqJ30Q8ZLfYLqFZEEX+G9MGbgw psQfeSCZKh6PG8aKavAmh+/4NkFamAvExmA993Igbq5iOTwjwupcyYVE+dge9LuqMFmz pUsveOMaC7JSC7+0OD9jLqGhMxebntZRUCv/8StBXivBbAPRRVrt9ICvtRCSDMezcgoj O5FQ== MIME-Version: 1.0 Received: by 10.50.53.193 with SMTP id d1mr12122482igp.69.1355282021894; Tue, 11 Dec 2012 19:13:41 -0800 (PST) Received: by 10.64.133.201 with HTTP; Tue, 11 Dec 2012 19:13:41 -0800 (PST) In-Reply-To: <755AC840-35B0-4896-8DAE-DB2418A293A4@thelastpickle.com> References: <755AC840-35B0-4896-8DAE-DB2418A293A4@thelastpickle.com> Date: Wed, 12 Dec 2012 11:13:41 +0800 Message-ID: Subject: Re: Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host) From: "dong.yajun" To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d04339b1c7c49ad04d09f2fae X-Virus-Checked: Checked by ClamAV on apache.org --f46d04339b1c7c49ad04d09f2fae Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks aaron. more information is I can read data correctly using Aqulies with Local_Quorum. I just check the system.log which is normal on 172.16.74.31 and the number of RPC time out is 10s. the client exception occured on 2012-12-05. and the all log on server on 2012-12-05 were: WARN [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.09080883864721297 INFO [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (line 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'Log') liveRatio is 1.0 (just-counted was 1.0). calculation took 4ms for 256 columns INFO [pool-1-thread-1] 2012-12-05 17:48:25,988 Memtable.java (line 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite') liveRatio is 2.780009341429239 (just-counted was 2.780009341429239). calculation took 1ms for 52 columns INFO [pool-1-thread-1] 2012-12-05 17:48:27,944 Memtable.java (line 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite') liveRatio is 2.780009341429239 (just-counted was 2.3128462147190714). calculation took 1ms for 75 columns INFO [CompactionExecutor:153] 2012-12-05 18:11:27,718 AutoSavingCache.java (line 269) Saved OpsCenter-rollups60-KeyCache (18 items) in 2 ms INFO [COMMIT-LOG-WRITER] 2012-12-05 20:31:10,025 CommitLogSegment.java (line 60) Creating new commitlog segment /data/cassandra/commitlog/CommitLog-1354768270025.log INFO [ScheduledTasks:1] 2012-12-05 21:47:38,352 GCInspector.java (line 123) GC for ParNew: 437 ms for 1 collections, 1163185072 used; max is 8375238656 INFO [CompactionExecutor:163] 2012-12-05 22:11:27,679 AutoSavingCache.java (line 269) Saved APIPortal-WebSite-KeyCache (1 items) in 2 ms On Wed, Dec 12, 2012 at 5:45 AM, aaron morton wrot= e: > > Caused by: TimedOutException() > Means the nodes involved in the request did not return to the co ordinato= r > before the rpc_timeout expired. > > Check the logs on the servers to see if they are overloaded and dropping > messages. > > Also check that you are not asking for too much data. > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 11/12/2012, at 10:13 PM, dong.yajun wrote: > > > hi list, > > I am using Cassandra with 3 data centers, each DC has more than 10 node= s. > > > > the schema for a keyspace: > > {DC1:3, DC2:3, DC3:3} > > > > now, I put some rows using hector with CL Local_Quorum in DC1 =EF=BC=8C= and then > I get a row with the same CL Local_Quorum in DC1=EF=BC=8Csome exceptions = were > occured: > > > > Cassandra with dsc-1.0.5, and Hector with 1.1-2. > > > > 2012-12-05 21:26:49,667 - WARN [pool-1-thread-3:JCLLoggerAdapter@379] - > Could not fullfill request on this host CassandraClient172.16.74.31:9160-= 1 > > 2012-12-05 21:26:49,668 - WARN [pool-1-thread-3:JCLLoggerAdapter@437] - > Exception: > > me.prettyprint.hector.api.exceptions.HTimedOutException: > TimedOutException() > > at > me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(Excep= tionsTranslatorImpl.java:35) > > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceSe= rviceImpl.java:163) > > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceSe= rviceImpl.java:145) > > at > me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.= java:103) > > at > me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailove= r(HConnectionManager.java:258) > > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(= KeyspaceServiceImpl.java:131) > > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(Keysp= aceServiceImpl.java:167) > > at > me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspa= ce(ThriftRangeSlicesQuery.java:66) > > at > me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspa= ce(ThriftRangeSlicesQuery.java:62) > > at > me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndM= easure(KeyspaceOperationCallback.java:20) > > at > me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeysp= ace.java:85) > > at > me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(Thri= ftRangeSlicesQuery.java:61) > > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolS= erver.java:176) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Sourc= e) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > > at java.lang.Thread.run(Unknown Source) > > Caused by: TimedOutException() > > at > org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassan= dra.java:12830) > > at > org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassan= dra.java:762) > > at > org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.j= ava:734) > > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceSe= rviceImpl.java:151) > > ... 19 more > > > > any suggestion with this issue? thanks. > > > > > > > > --=20 *Ric Dong * Newegg Ecommerce, MIS department --f46d04339b1c7c49ad04d09f2fae Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks aaron.=C2=A0

more information is I can read data = =C2=A0correctly=C2=A0using Aqulies=C2=A0with Local_Quorum.=C2=A0
=

I just check the system.log which is normal on=C2=A0172= .16.74.31 and the number of RPC time out is 10s. the client exception occur= ed on 2012-12-05.=C2=A0

and the all log on server on 2012-12-05 were:

=C2=A0WARN [pool-1-thread-1] 2012-12-05 11:17:10,974 M= emtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0= 9080883864721297
=C2=A0INFO [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (li= ne 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'Log') l= iveRatio is 1.0 (just-counted was 1.0). =C2=A0calculation took 4ms for 256 = columns
=C2=A0INFO [pool-1-thread-1] 2012-12-05 17:48:25,988 Memtable.java (li= ne 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite'= ;) liveRatio is 2.780009341429239 (just-counted was 2.780009341429239). =C2= =A0calculation took 1ms for 52 columns
=C2=A0INFO [pool-1-thread-1] 2012-12-05 17:48:27,944 Memtable.java (li= ne 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite'= ;) liveRatio is 2.780009341429239 (just-counted was 2.3128462147190714). = =C2=A0calculation took 1ms for 75 columns
=C2=A0INFO [CompactionExecutor:153] 2012-12-05 18:11:27,718 AutoSaving= Cache.java (line 269) Saved OpsCenter-rollups60-KeyCache (18 items) in 2 ms=
=C2=A0INFO [COMMIT-LOG-WRITER] 2012-12-05 20:31:10,025 CommitLog= Segment.java (line 60) Creating new commitlog segment /data/cassandra/commi= tlog/CommitLog-1354768270025.log
=C2=A0INFO [ScheduledTasks:1] 2012-12-05 21:47:38,352 GCInspector.java= (line 123) GC for ParNew: 437 ms for 1 collections, 1163185072 used; max i= s 8375238656
=C2=A0INFO [CompactionExecutor:163] 2012-12-05 22:11= :27,679 AutoSavingCache.java (line 269) Saved APIPortal-WebSite-KeyCache (1= items) in 2 ms

On Wed, Dec 12, 2012 at 5:45 AM, aaron morto= n <aaron@thelastpickle.com> wrote:
> Caused by: TimedOutException()
Means the nodes involved in the request did not return to the co ordinator = before the rpc_timeout expired.

Check the logs on the servers to see if they are overloaded and dropping me= ssages.

Also check that you are not asking for too much data.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thela= stpickle.com

On 11/12/2012, at 10:13 PM, dong.yajun <dongtalk@gmail.com> wrote:

> hi list,
> I am using Cassandra with 3 data centers, each DC has more than 10 nod= es.
>
> the schema for a keyspace:
> {DC1:3, DC2:3, DC3:3}
>
> now, I put some rows using hector with CL Local_Quorum in DC1 =EF=BC= =8Cand then I get a row with the same CL Local_Quorum in DC1=EF=BC=8Csome e= xceptions were occured:
>
> Cassandra with dsc-1.0.5, and Hector with 1.1-2.
>
> 2012-12-05 21:26:49,667 - WARN [pool-1-thread-3:JCLLoggerAdapter@379] = - Could not fullfill request on this host CassandraClient172.16.74.31:9160-= 1
> 2012-12-05 21:26:49,668 - WARN [pool-1-thread-3:JCLLoggerAdapter@437] = - Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutExcep= tion()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate= (ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(Keys= paceServiceImpl.java:163)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(Keys= paceServiceImpl.java:145)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Oper= ation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithF= ailover(HConnectionManager.java:258)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFai= lover(KeyspaceServiceImpl.java:131)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices= (KeyspaceServiceImpl.java:167)
> at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doIn= Keyspace(ThriftRangeSlicesQuery.java:66)
> at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doIn= Keyspace(ThriftRangeSlicesQuery.java:62)
> at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspa= ceAndMeasure(KeyspaceOperationCallback.java:20)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(Executin= gKeyspace.java:85)
> at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execut= e(ThriftRangeSlicesQuery.java:61)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPo= olServer.java:176)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Sour= ce)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<= br> > at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(= Cassandra.java:12830)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(= Cassandra.java:762)
> at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassa= ndra.java:734)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(Keys= paceServiceImpl.java:151)
> ... 19 more
>
> any suggestion with this issue? thanks.
>
>
>




--
= Ric Dong=C2=A0
Newegg Ecommerce, MIS department=C2=A0


--f46d04339b1c7c49ad04d09f2fae--