Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0995AD28E for ; Wed, 12 Dec 2012 07:12:53 +0000 (UTC) Received: (qmail 19395 invoked by uid 500); 12 Dec 2012 07:12:50 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 19262 invoked by uid 500); 12 Dec 2012 07:12:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 19239 invoked by uid 99); 12 Dec 2012 07:12:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 07:12:49 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dongtalk@gmail.com designates 209.85.210.172 as permitted sender) Received: from [209.85.210.172] (HELO mail-ia0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Dec 2012 07:12:43 +0000 Received: by mail-ia0-f172.google.com with SMTP id z13so608935iaz.17 for ; Tue, 11 Dec 2012 23:12:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=LPimE/yqVNXrWpgkNv/UX2V/jV+RNLsVojnA1Ww++Pw=; b=B3+Jua7DDxA0quY5GvQrXtsqOOaCxoJ+I/ZNTrb0+zuOw9UveMH+yn36R/Vm/5l6Jx M20Zzb64cPw0qBiBfV2j9qkJfFfM7sdCc/1oRlz+oFOU3irRIMwO7V3B1ccbH2119+wa BwTqbaQJ+FueodM6mprlDBnFa5GHBmAKvoCPIB05IWPAaGvImqMMubCmeNxX6n7Ngms3 GHmpjHMkKzBi31xS+8cz0+zJQCNLBiWl8HLzqQsqB0W3VZ3VF20qWOXyjlKoVKwf8uJh 7D3vmp8/xVo/jGMEczZgtVeFmZHoz9sBDyufkqp5Pagm9dKtv4kz56/GoFGDCHMyvRAW tJhA== MIME-Version: 1.0 Received: by 10.50.15.138 with SMTP id x10mr12550507igc.16.1355296343315; Tue, 11 Dec 2012 23:12:23 -0800 (PST) Received: by 10.64.133.201 with HTTP; Tue, 11 Dec 2012 23:12:23 -0800 (PST) In-Reply-To: References: <755AC840-35B0-4896-8DAE-DB2418A293A4@thelastpickle.com> Date: Wed, 12 Dec 2012 15:12:23 +0800 Message-ID: Subject: Re: Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host) From: "dong.yajun" To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=14dae9340bef1bd51204d0a28521 X-Virus-Checked: Checked by ClamAV on apache.org --14dae9340bef1bd51204d0a28521 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable hi aaron. There is no problem with get_slices but get_range_slices failed. the way I use this method is: sliceRangeQuery.setFamily(family); sliceRangeQuery.setKeys(rowkey,rowkey); sliceRangeQuery.setRange(Character.Min_Value, Character.Max_Value, false, Integer.Max_Value); sliceRangeQuery.setRowCount(1); current family has only abort 10000 rows, each row has ~10columns, the data abort 25k. On Wed, Dec 12, 2012 at 11:13 AM, dong.yajun wrote: > Thanks aaron. > > more information is I can read data correctly using Aqulies with > Local_Quorum. > > I just check the system.log which is normal on 172.16.74.31 and the numbe= r > of RPC time out is 10s. the client exception occured on 2012-12-05. > > and the all log on server on 2012-12-05 were: > > WARN [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (line 169) > setting live ratio to minimum of 1.0 instead of 0.09080883864721297 > INFO [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (line 179) > CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'Log') liveRatio is 1.0 > (just-counted was 1.0). calculation took 4ms for 256 columns > INFO [pool-1-thread-1] 2012-12-05 17:48:25,988 Memtable.java (line 179) > CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite') liveRatio is > 2.780009341429239 (just-counted was 2.780009341429239). calculation took > 1ms for 52 columns > INFO [pool-1-thread-1] 2012-12-05 17:48:27,944 Memtable.java (line 179) > CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite') liveRatio is > 2.780009341429239 (just-counted was 2.3128462147190714). calculation too= k > 1ms for 75 columns > INFO [CompactionExecutor:153] 2012-12-05 18:11:27,718 > AutoSavingCache.java (line 269) Saved OpsCenter-rollups60-KeyCache (18 > items) in 2 ms > INFO [COMMIT-LOG-WRITER] 2012-12-05 20:31:10,025 CommitLogSegment.java > (line 60) Creating new commitlog segment > /data/cassandra/commitlog/CommitLog-1354768270025.log > INFO [ScheduledTasks:1] 2012-12-05 21:47:38,352 GCInspector.java (line > 123) GC for ParNew: 437 ms for 1 collections, 1163185072 used; max is > 8375238656 > INFO [CompactionExecutor:163] 2012-12-05 22:11:27,679 > AutoSavingCache.java (line 269) Saved APIPortal-WebSite-KeyCache (1 items= ) > in 2 ms > > On Wed, Dec 12, 2012 at 5:45 AM, aaron morton wr= ote: > >> > Caused by: TimedOutException() >> Means the nodes involved in the request did not return to the co >> ordinator before the rpc_timeout expired. >> >> Check the logs on the servers to see if they are overloaded and dropping >> messages. >> >> Also check that you are not asking for too much data. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 11/12/2012, at 10:13 PM, dong.yajun wrote: >> >> > hi list, >> > I am using Cassandra with 3 data centers, each DC has more than 10 >> nodes. >> > >> > the schema for a keyspace: >> > {DC1:3, DC2:3, DC3:3} >> > >> > now, I put some rows using hector with CL Local_Quorum in DC1 =EF=BC= =8Cand then >> I get a row with the same CL Local_Quorum in DC1=EF=BC=8Csome exceptions= were >> occured: >> > >> > Cassandra with dsc-1.0.5, and Hector with 1.1-2. >> > >> > 2012-12-05 21:26:49,667 - WARN [pool-1-thread-3:JCLLoggerAdapter@379] >> - Could not fullfill request on this host CassandraClient172.16.74.31:91= 60-1 >> > 2012-12-05 21:26:49,668 - WARN [pool-1-thread-3:JCLLoggerAdapter@437] >> - Exception: >> > me.prettyprint.hector.api.exceptions.HTimedOutException: >> TimedOutException() >> > at >> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(Exce= ptionsTranslatorImpl.java:35) >> > at >> me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceS= erviceImpl.java:163) >> > at >> me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceS= erviceImpl.java:145) >> > at >> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation= .java:103) >> > at >> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailov= er(HConnectionManager.java:258) >> > at >> me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover= (KeyspaceServiceImpl.java:131) >> > at >> me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(Keys= paceServiceImpl.java:167) >> > at >> me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeysp= ace(ThriftRangeSlicesQuery.java:66) >> > at >> me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeysp= ace(ThriftRangeSlicesQuery.java:62) >> > at >> me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAnd= Measure(KeyspaceOperationCallback.java:20) >> > at >> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeys= pace.java:85) >> > at >> me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(Thr= iftRangeSlicesQuery.java:61) >> > >> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPool= Server.java:176) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown >> Source) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) >> > at java.lang.Thread.run(Unknown Source) >> > Caused by: TimedOutException() >> > at >> org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassa= ndra.java:12830) >> > at >> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassa= ndra.java:762) >> > at >> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.= java:734) >> > at >> me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceS= erviceImpl.java:151) >> > ... 19 more >> > >> > any suggestion with this issue? thanks. >> > >> > >> > >> >> > > > -- > *Ric Dong * > Newegg Ecommerce, MIS department > > > --=20 *Ric Dong * Newegg Ecommerce, MIS department --14dae9340bef1bd51204d0a28521 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable hi aaron.=C2=A0

There is no problem with get_slices but = get_range_slices failed. the way I use this method is:=C2=A0

=
sliceRangeQuery.setFamily(family);
sliceRangeQuery.set= Keys(rowkey,rowkey);
sliceRangeQuery.setRange(Character.Min_Value, Character.Max_Value, fal= se, Integer.Max_Value);
sliceRangeQuery.setRowCount(1);=C2=A0

current family has only abort 10000 rows, each row ha= s ~10columns, the data abort 25k.=C2=A0


On Wed, Dec 12, 2012 at = 11:13 AM, dong.yajun <dongtalk@gmail.com> wrote:
Thanks aaron.=C2=A0

more information is I can read data = =C2=A0correctly=C2=A0using Aqulies=C2=A0with Local_Quorum.=C2=A0
=

I just check the system.log which is normal on=C2=A0172= .16.74.31 and the number of RPC time out is 10s. the client exception occur= ed on 2012-12-05.=C2=A0

and the all log on server on 2012-12-05 were:

=C2=A0WARN [pool-1-thread-1] 2012-12-05 11:17:10,974 M= emtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0= 9080883864721297
=C2=A0INFO [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (li= ne 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'Log') l= iveRatio is 1.0 (just-counted was 1.0). =C2=A0calculation took 4ms for 256 = columns
=C2=A0INFO [pool-1-thread-1] 2012-12-05 17:48:25,988 Memtable.java (li= ne 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite'= ;) liveRatio is 2.780009341429239 (just-counted was 2.780009341429239). =C2= =A0calculation took 1ms for 52 columns
=C2=A0INFO [pool-1-thread-1] 2012-12-05 17:48:27,944 Memtable.java (li= ne 179) CFS(Keyspace=3D'APIPortal', ColumnFamily=3D'WebSite'= ;) liveRatio is 2.780009341429239 (just-counted was 2.3128462147190714). = =C2=A0calculation took 1ms for 75 columns
=C2=A0INFO [CompactionExecutor:153] 2012-12-05 18:11:27,718 AutoSaving= Cache.java (line 269) Saved OpsCenter-rollups60-KeyCache (18 items) in 2 ms=
=C2=A0INFO [COMMIT-LOG-WRITER] 2012-12-05 20:31:10,025 CommitLog= Segment.java (line 60) Creating new commitlog segment /data/cassandra/commi= tlog/CommitLog-1354768270025.log
=C2=A0INFO [ScheduledTasks:1] 2012-12-05 21:47:38,352 GCInspector.java= (line 123) GC for ParNew: 437 ms for 1 collections, 1163185072 used; max i= s 8375238656
=C2=A0INFO [CompactionExecutor:163] 2012-12-05 22:11= :27,679 AutoSavingCache.java (line 269) Saved APIPortal-WebSite-KeyCache (1= items) in 2 ms

On Wed, Dec 12, 2012 at 5:45 AM, aaron morto= n <aaron@thelastpickle.com> wrote:
> Caused by: TimedOutException()
Means the nodes involved in the request did not return to the co ordinator = before the rpc_timeout expired.

Check the logs on the servers to see if they are overloaded and dropping me= ssages.

Also check that you are not asking for too much data.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thela= stpickle.com

On 11/12/2012, at 10:13 PM, dong.yajun <dongtalk@gmail.com> wrote:

> hi list,
> I am using Cassandra with 3 data centers, each DC has more than 10 nod= es.
>
> the schema for a keyspace:
> {DC1:3, DC2:3, DC3:3}
>
> now, I put some rows using hector with CL Local_Quorum in DC1 =EF=BC= =8Cand then I get a row with the same CL Local_Quorum in DC1=EF=BC=8Csome e= xceptions were occured:
>
> Cassandra with dsc-1.0.5, and Hector with 1.1-2.
>
> 2012-12-05 21:26:49,667 - WARN [pool-1-thread-3:JCLLoggerAdapter@379] = - Could not fullfill request on this host CassandraClient172.16.74.31:9160-= 1
> 2012-12-05 21:26:49,668 - WARN [pool-1-thread-3:JCLLoggerAdapter@437] = - Exception:
> me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutExcep= tion()
> at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate= (ExceptionsTranslatorImpl.java:35)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(Keys= paceServiceImpl.java:163)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(Keys= paceServiceImpl.java:145)
> at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Oper= ation.java:103)
> at me.prettyprint.cassandra.connection.HConnectionManager.operateWithF= ailover(HConnectionManager.java:258)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFai= lover(KeyspaceServiceImpl.java:131)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices= (KeyspaceServiceImpl.java:167)
> at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doIn= Keyspace(ThriftRangeSlicesQuery.java:66)
> at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doIn= Keyspace(ThriftRangeSlicesQuery.java:62)
> at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspa= ceAndMeasure(KeyspaceOperationCallback.java:20)
> at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(Executin= gKeyspace.java:85)
> at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execut= e(ThriftRangeSlicesQuery.java:61)
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPo= olServer.java:176)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Sour= ce)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<= br> > at java.lang.Thread.run(Unknown Source)
> Caused by: TimedOutException()
> at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(= Cassandra.java:12830)
> at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(= Cassandra.java:762)
> at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassa= ndra.java:734)
> at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(Keys= paceServiceImpl.java:151)
> ... 19 more
>
> any suggestion with this issue? thanks.
>
>
>




<= /div>--
Ric Dong=C2=A0
New= egg Ecommerce, MIS department=C2=A0





--
Ric Dong=C2=A0
Newegg Ecommerce, MIS department=C2=A0


--14dae9340bef1bd51204d0a28521--