Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 40888CD77 for ; Wed, 30 May 2012 21:41:41 +0000 (UTC) Received: (qmail 21024 invoked by uid 500); 30 May 2012 21:41:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 21002 invoked by uid 500); 30 May 2012 21:41:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 20993 invoked by uid 99); 30 May 2012 21:41:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 May 2012 21:41:38 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a91.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 30 May 2012 21:41:30 +0000 Received: from homiemail-a91.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a91.g.dreamhost.com (Postfix) with ESMTP id F02A8AE07A for ; Wed, 30 May 2012 14:41:06 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=4p4WsWMOUy 1qFZRCwkw22fKCKrHL0RE/4GKVM5OObEeEIhEmWuErM547/MuwT8phvomYOxHqkd XDErMj1fVLG1eifMv3B2Whu7rfS6OUqbvVjDNGLzGk7chLKbi1TZQjv4zUJEJqKb YX3/wVGg0Kj/Haak5Dhzmig8LrvTKU+nw= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=fMPWZC3N79aEFnX0 8WMdwgF3HsQ=; b=dlFZbNeVrKB5u1l/5ncINyr6innzBfKQvG88sNDBr2Zsno40 ZBrrz5JUv5BeQQkgNEUDPC8lhim6x0wvlWSeL3LR7J4NDyoeJYe1ukZA/dcchKgS yq+fcosDlU9Q+Yt/g/P++pyUN7ShhbrX02EU0pPZY0PGaZXetNsPR4tAu50= Received: from [172.16.1.4] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a91.g.dreamhost.com (Postfix) with ESMTPSA id 0C67CAE075 for ; Wed, 30 May 2012 14:41:05 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/alternative; boundary="Apple-Mail=_7C6F7335-320E-4A4D-B0C5-612589D467B4" Subject: Re: unknown exception with hector Date: Thu, 31 May 2012 09:41:02 +1200 In-Reply-To: <4FBED991.2050405@syncopated.net> To: user@cassandra.apache.org References: <4FBBCE5F.9040509@syncopated.net> <43D18FFE-E9A8-4AD9-AF3C-AA7B5C1DFF2C@thelastpickle.com> <4FBDCC97.2030705@syncopated.net> <4FBED991.2050405@syncopated.net> Message-Id: X-Mailer: Apple Mail (2.1278) --Apple-Mail=_7C6F7335-320E-4A4D-B0C5-612589D467B4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > i'm not sure if using framed transport is an option with hector. = http://hector-client.github.com/hector//source/content/API/core/0.8.0-2/me= /prettyprint/cassandra/service/CassandraHostConfigurator.html#setUseThrift= FramedTransport(boolean) > what should i be in the logs looking for to find the cause of these = dropped reads? These look like transport errors. If something is happening on the = server side it will be logged at ERROR level.=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 25/05/2012, at 1:00 PM, Deno Vichas wrote: > i'm not sure if using framed transport is an option with hector. >=20 > what should i be in the logs looking for to find the cause of these = dropped reads? >=20 > thanks, >=20 > On 5/24/2012 3:04 AM, aaron morton wrote: >>=20 >> Dropped read messages occur when the node could not process a read = task within rpc_timeout. It generally means the cluster has been = overwhelmed at some point: too many requests, to much GC, compaction = hurting, etc.=20 >>=20 >> Check the server side logs for errors but I doubt it is related to = the call stack below.=20 >>=20 >> Have you confirmed that you are using framed transport on the client? >>=20 >> Cheers >>=20 >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >>=20 >> On 24/05/2012, at 5:52 PM, Deno Vichas wrote: >>=20 >>> i've notice the my nodes seem to have a large (?, not really sure = what acceptable numbers are) read dropped count from = tpstats. could they be related?=20 >>>=20 >>> On 5/23/2012 2:55 AM, aaron morton wrote: >>>>=20 >>>> No sure but=20 >>>>=20 >>>>> at = org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.jav= a:132) >>>>=20 >>>> Looks like the client is not using framed transport. The server = defaults to framed. >>>>=20 >>>> Cheers >>>>=20 >>>> ----------------- >>>> Aaron Morton >>>> Freelance Developer >>>> @aaronmorton >>>> http://www.thelastpickle.com >>>>=20 >>>> On 23/05/2012, at 5:35 AM, Deno Vichas wrote: >>>>=20 >>>>> could somebody clue me in to the cause of this exception? i see = these randomly. >>>>>=20 >>>>> AnalyzerService-2 2012-05-22 13:28:00,385 :: WARN = cassandra.connection.HConnectionManager - Exception: >>>>> me.prettyprint.hector.api.exceptions.HectorTransportException: = org.apache.thrift.transport.TTransportException >>>>> at = me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(Except= ionsTranslatorImpl.java:39) >>>>> at = me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceSe= rviceImpl.java:851) >>>>> at = me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceSe= rviceImpl.java:840) >>>>> at = me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.j= ava:99) >>>>> at = me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover= (HConnectionManager.java:243) >>>>> at = me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(K= eyspaceServiceImpl.java:131) >>>>> at = me.prettyprint.cassandra.service.KeyspaceServiceImpl.getColumn(KeyspaceSer= viceImpl.java:857) >>>>> at = me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(Thr= iftColumnQuery.java:57) >>>>> at = me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(Thr= iftColumnQuery.java:52) >>>>> at = me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMe= asure(KeyspaceOperationCallback.java:20) >>>>> at = me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspa= ce.java:85) >>>>> at = me.prettyprint.cassandra.model.thrift.ThriftColumnQuery.execute(ThriftColu= mnQuery.java:51) >>>>> at = com.stocktouch.dao.StockDaoImpl.getHistorical(StockDaoImpl.java:365) >>>>> at = com.stocktouch.dao.StockDaoImpl.getHistoricalQuote(StockDaoImpl.java:433) >>>>> at = com.stocktouch.service.StockHistoryServiceImpl.getHistoricalQuote(StockHis= toryServiceImpl.java:480) >>>>> at = com.stocktouch.service.AnalyzerServiceImpl.getClose(AnalyzerServiceImpl.ja= va:180) >>>>> at = com.stocktouch.service.AnalyzerServiceImpl.calcClosingPrices(AnalyzerServi= ceImpl.java:90) >>>>> at = com.stocktouch.service.AnalyzerServiceImpl.nightlyRollup(AnalyzerServiceIm= pl.java:66) >>>>> at = com.stocktouch.service.AnalyzerServiceImpl$2.run(AnalyzerServiceImpl.java:= 55) >>>>> at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886) >>>>> at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> Caused by: org.apache.thrift.transport.TTransportException >>>>> at = org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.jav= a:132) >>>>> at = org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>> at = org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.ja= va:129) >>>>> at = org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:10= 1) >>>>> at = org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>> at = org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:37= 8) >>>>> at = org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:29= 7) >>>>> at = org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtoco= l.java:204) >>>>> at = org.apache.cassandra.thrift.Cassandra$Client.recv_get(Cassandra.java:509) >>>>> at = org.apache.cassandra.thrift.Cassandra$Client.get(Cassandra.java:492) >>>>> at = me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceSe= rviceImpl.java:846) >>>>> ... 20 more >>>>>=20 >>>>>=20 >>>>> thanks, >>>>> deno >>>>>=20 >>>>>=20 >>>>=20 >>>=20 >>=20 >=20 --Apple-Mail=_7C6F7335-320E-4A4D-B0C5-612589D467B4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
i'm not sure if using framed transport is an option = with hector.
http://hector-client.github.com/hector//= source/content/API/core/0.8.0-2/me/prettyprint/cassandra/service/Cassandra= HostConfigurator.html#setUseThriftFramedTransport(boolean)

what should i be in the logs looking for to find the = cause of these dropped reads?
These look like = transport errors. If something is happening on the server side it will = be logged at ERROR = level. 

Cheers


http://www.thelastpickle.com

On 25/05/2012, at 1:00 PM, Deno Vichas wrote:

=20 =20
i'm not sure if using framed transport is an option with hector.

what should i be in the logs looking for to find the cause of these dropped reads?

thanks,

On 5/24/2012 3:04 AM, aaron morton wrote:
Dropped read messages occur when the node could not process a read task within rpc_timeout. It generally means the cluster has been overwhelmed at some point:  too many = requests, to much GC, compaction hurting, etc. 

Check the server side logs for errors but I doubt it is related to the call stack below. 

Have you confirmed that you are using framed transport on the client?

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 24/05/2012, at 5:52 PM, Deno Vichas wrote:

i've notice the = my nodes seem to have a large (?, not really sure what acceptable numbers are) read dropped count from tpstats.  could they be related?

On 5/23/2012 2:55 AM, aaron morton wrote:
No sure but 

       at = org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.jav= a:132)

Looks like the client is not using framed transport. The server defaults to framed.

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton

On 23/05/2012, at 5:35 AM, Deno Vichas = wrote:

could somebody clue me in to the cause of this exception?  i see these = randomly.

AnalyzerService-2 2012-05-22 13:28:00,385 :: WARN =  cassandra.connection.HConnectionManager  - Exception:
= me.prettyprint.hector.api.exceptions.HectorTransportException: org.apache.thrift.transport.TTransportException
       at = me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(Except= ionsTranslatorImpl.java:39)
       at = me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceSe= rviceImpl.java:851)
       at = me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceSe= rviceImpl.java:840)
       at = me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.j= ava:99)
       at = me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover= (HConnectionManager.java:243)
       at = me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(K= eyspaceServiceImpl.java:131)
       at = me.prettyprint.cassandra.service.KeyspaceServiceImpl.getColumn(KeyspaceSer= viceImpl.java:857)
       at = me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(Thr= iftColumnQuery.java:57)
       at = me.prettyprint.cassandra.model.thrift.ThriftColumnQuery$1.doInKeyspace(Thr= iftColumnQuery.java:52)
       at = me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMe= asure(KeyspaceOperationCallback.java:20)
       at = me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspa= ce.java:85)
       at = me.prettyprint.cassandra.model.thrift.ThriftColumnQuery.execute(ThriftColu= mnQuery.java:51)
       at = com.stocktouch.dao.StockDaoImpl.getHistorical(StockDaoImpl.java:365)
       at = com.stocktouch.dao.StockDaoImpl.getHistoricalQuote(StockDaoImpl.java:433)<= br>        at = com.stocktouch.service.StockHistoryServiceImpl.getHistoricalQuote(StockHis= toryServiceImpl.java:480)
       at = com.stocktouch.service.AnalyzerServiceImpl.getClose(AnalyzerServiceImpl.ja= va:180)
       at = com.stocktouch.service.AnalyzerServiceImpl.calcClosingPrices(AnalyzerServi= ceImpl.java:90)
       at = com.stocktouch.service.AnalyzerServiceImpl.nightlyRollup(AnalyzerServiceIm= pl.java:66)
       at = com.stocktouch.service.AnalyzerServiceImpl$2.run(AnalyzerServiceImpl.java:= 55)
       at = java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.= java:886)
       at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :908)
       at java.lang.Thread.run(Thread.java:662)
Caused by: = org.apache.thrift.transport.TTransportException
       at = org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.jav= a:132)
       at = org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
       at = org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.ja= va:129)
       at = org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:10= 1)
       at = org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
       at = org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:37= 8)
       at = org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:29= 7)
       at = org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtoco= l.java:204)
       at = org.apache.cassandra.thrift.Cassandra$Client.recv_get(Cassandra.java:509)<= br>        at = org.apache.cassandra.thrift.Cassandra$Client.get(Cassandra.java:492)
       at = me.prettyprint.cassandra.service.KeyspaceServiceImpl$23.execute(KeyspaceSe= rviceImpl.java:846)
       ... = 20 more


thanks,
deno







= --Apple-Mail=_7C6F7335-320E-4A4D-B0C5-612589D467B4--