Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EA1C31065F for ; Thu, 16 Jan 2014 16:30:43 +0000 (UTC) Received: (qmail 13796 invoked by uid 500); 16 Jan 2014 16:30:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13736 invoked by uid 500); 16 Jan 2014 16:30:40 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13728 invoked by uid 99); 16 Jan 2014 16:30:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jan 2014 16:30:40 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of tyler@datastax.com designates 209.85.217.179 as permitted sender) Received: from [209.85.217.179] (HELO mail-lb0-f179.google.com) (209.85.217.179) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jan 2014 16:30:34 +0000 Received: by mail-lb0-f179.google.com with SMTP id l4so483399lbv.38 for ; Thu, 16 Jan 2014 08:30:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=sbGNTaXQZMdcg2kSJCFVtzG9aV7K7VLxb72f8EABAiE=; b=NjIl5CP2Rler+soP6xlRiqq7/K75R28tXiFwmPV5TVPEH/+pK3EAL0lr0vCjEYVQlr +0IRZiPhewhhFB9CaP8pzHVG6Cx1bDtURH2/N6gGrhMWWKqHGTRrQpgRN/OJoPpKPMp9 L/GEMGUukkC/zV40mhIA4pH9hF3vY5Dat6F6mc2NiD2YP+Dssomp6Rqok/dPc5vig2L/ jLyw2aovqt0RNe9/9EoyeIf6TKA44ouOlSG9Vo/bzXuxP1TTZb0J/23mIl2sb713H13F NDIptLD6k4iwXkooOMGqK+EahmbjNpY94i1GpDPfr7PkX2WXkDqz/0CK5b44hF6Uh39U Tlbw== X-Gm-Message-State: ALoCoQlfiIqEpE2/rkXIuYSDiG3oeCfWObIHlXhRkNKeOTCdhdE+MmlyBY8Eo6RfGLtk3zi9zoou X-Received: by 10.152.180.35 with SMTP id dl3mr619887lac.66.1389889811053; Thu, 16 Jan 2014 08:30:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.135.230 with HTTP; Thu, 16 Jan 2014 08:29:50 -0800 (PST) In-Reply-To: References: From: Tyler Hobbs Date: Thu, 16 Jan 2014 10:29:50 -0600 Message-ID: Subject: Re: possible problem with multi-node cluster To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11345a7a77105404f018f01a X-Virus-Checked: Checked by ClamAV on apache.org --001a11345a7a77105404f018f01a Content-Type: text/plain; charset=ISO-8859-1 Do you see any errors in the system.log for DSE/Cassandra? On Thu, Jan 16, 2014 at 9:19 AM, Amalrik Maia wrote: > Hi guys, > i have some issues with an application on a cassandra multinode cluster. > I need help to understand what is going on. > > here is the stack trace of my application. > > 2014-01-16 12:33:14,176 - root - ERROR - ------------------------------------------------------------ > 2014-01-16 12:33:14,176 - root - ERROR - Task finished processing with exception, message will be kept in the queue > 2014-01-16 12:33:14,180 - pycassa.pool - INFO - Connection 42979792 (cassandra.*******.com:9160) in pool 42966096 failed: > 2014-01-16 12:33:14,182 - pycassa.pool - INFO - Connection 42979408 (cassandra.*******.com:9160) in pool 42966096 failed: > 2014-01-16 12:33:14,182 - pycassa.pool - INFO - Connection 42966864 (cassandra.*******.com:9160) in pool 42966096 failed: > 2014-01-16 12:33:14,183 - root - ERROR - Exception in user code: > 2014-01-16 12:33:14,183 - root - ERROR - ------------------------------------------------------------ > 2014-01-16 12:33:14,183 - root - ERROR - (, MaximumRetryException('Retried 1 times. Last failure was EOFError: ',), ) > 2014-01-16 12:33:14,183 - root - ERROR - Traceback (most recent call last): > > File "acme.py", line 20, in task_run > > run_method_result = run_method(message.get_body()) > > File "/opt/acme/acme/acme/apps/stats_updater/stats_updater.py", line 141, in acme_run > > StatsUpdater.get().run(params) > > File "/opt/acme/acme/acme/apps/stats_updater/stats_updater.py", line 94, in run > > result = StatisticsService.get().increment_counters(cookie, counter_type, obj_id, pdate, 1, tag_name) > > File "/opt/acme/local/lib/python2.7/site-packages/s1audservice/statistics_service.py", line 41, in increment_counters > > self.do_increment_counter(False, FrequencyCriteria.HOUR, counter_type_tag, obj_id, date, value) > > File "/opt/acme/local/lib/python2.7/site-packages/s1audservice/statistics_service.py", line 389, in do_increment_counter > > return self.increment_counter(counter_type_tag, f_date, nosql_id, cf_to_update, value_to_increment) > > File "/opt/acme/local/lib/python2.7/site-packages/s1audservice/statistics_service.py", line 397, in increment_counter > > pycassa.ConsistencyLevel.ALL) > > File "/opt/acme/local/lib/python2.7/site-packages/pycassa/columnfamily.py", line 1066, in add > > allow_retries=self._allow_retries) > > File "/opt/acme/local/lib/python2.7/site-packages/pycassa/pool.py", line 577, in execute > > return getattr(conn, f)(*args, **kwargs) > > File "/opt/acme/local/lib/python2.7/site-packages/pycassa/pool.py", line 148, in new_f > > (self._retry_count, exc.__class__.__name__, exc)) > MaximumRetryException: Retried 1 times. Last failure was EOFError: > > I'm using DataStax Enterprise version 3.2.2-1, with cassandra 1.2.12.2. > my application is written in python 2.7 and connects cassandra through > pycassa 1.10.0, my thrift version is 0.9.1 > > here is the code that connects cassandra: > > > __instances = {} > __CONNECTION_POOL_SIZE_PER_HOST = 20 > > def __init__(self, prefix): > if not prefix: > raise ValueError ("Invalid prefix for connection") > > self.ip_seeds = ConfigService.get().get_property(prefix + ".host") + ":"\ > + ConfigService.get().get_property(prefix + ".port") > self.cluster_name = ConfigService.get().get_property(prefix + ".cluster_name") > self.keyspace_name = ConfigService.get().get_property(prefix + ".keyspace") > self.conec_pool_name = ConfigService.get().get_property(prefix + ".pool_name") > self.con_pool = pycassa.ConnectionPool(self.keyspace_name, server_list=[self.ip_seeds], timeout=50) > > > > this is the code that triggers the exception: > > cf_to_update.add( > (counter_type_tag, long(str_date_formatted)), > nosql_id, > int(value), > None, > pycassa.ConsistencyLevel.ALL) > > > > This same code is running successfully without any errors with a single > node cassandra cluster and very close configurations: > python 2.7 apache cassandra 1.2.8, pycassa 1.10.0, thrift 0.9.1 > > any help would be greatly appreciated. > -- Tyler Hobbs DataStax --001a11345a7a77105404f018f01a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Do you see any errors in the system.log for DSE/Cassandra?=


On = Thu, Jan 16, 2014 at 9:19 AM, Amalrik Maia <amalrik@s1mbi0se.com.br<= /a>> wrote:
Hi = guys,
i have some issues with an application on a cassandra multi= node cluster.
I need help to understand what is going on.
=A0
he= re is the stack trace of my application.

2014-01-16 1=
2:33:14,176 - root - ERROR - -------------------------------------=
-----------------------


2014-01-16 12:33:14,176 - root - ERRO=
R - Task finished processing with exce=
ption, message will be kept in the queue


2014-01-16 12:33:14,180 - pycassa.poo=
l - INFO - Connection 42979792 (cassandra.******=
*.com:9160)<=
span style=3D"color:rgb(77,77,76)"> in pool 42966096 failed: 


2014-01-16 12:33:14,182 - pycassa.poo=
l - INFO - Connection 42979408 (cassandra.******=
*.com:9160)<=
span style=3D"color:rgb(77,77,76)"> in pool 42966096 failed: 


2014-01-16 12:33:14,182 - pycassa.poo=
l - INFO - Connection 42966864 (cassandra.*******.com:9160)<=
span style=3D"color:rgb(77,77,76)"> in pool 42966096 failed: 


2014-01-16 12:33:14,183 - root - ERRO=
R - Exception in user code:=



2014-01-16 12:33:14,183 - root - ERRO=
R - -----------------=
-------------------------------------------


2014-01-16 12:33:14,183 - root - ERRO=
R - (<class 'pycassa.pool.Maximu=
mRetryException'>,=
 MaximumRetryException('Retried 1 times. Last failure=
 was EOFError: ',), <traceback object at 0x28fe3b0>)


2014-01-16 12:33:14,183 - root - ERRO=
R - Traceback (most recent call <=
span style=3D"color:rgb(137,89,168)">last):


 =A0File "acme.py", line 20, in task_run


 =A0 =A0run_method_result =3D run_method(message.get_body())


 =A0File "/op=
t/acme/acme/acme/apps/stats_updater/stats_updater.py", line=
 141, in acme_run


 =A0 =A0StatsUpdater.get().run(=
params)


 =A0File "/op=
t/acme/acme/acme/apps/stats_updater/stats_updater.py", line=
 94, in run


 =A0 =A0result =3D =
StatisticsService.get().increment_counters(cookie, counter_type, obj_id, pdate, 1, tag_n=
ame)


 =A0File "/op=
t/acme/local/lib/python2.7/site-packages/s1audservice/statistics_service.py=
", line 41, in increment_counters


 =A0 =A0self.do_inc=
rement_counter(False, FrequencyCriteria.HOUR, counter_type_=
tag, obj_id, date, value)


 =A0File "/op=
t/acme/local/lib/python2.7/site-packages/s1audservice/statistics_service.py=
", line 389, in do_increment_counter


 =A0 =A0return se=
lf.increment_counter(=
counter_type_tag, f_d=
ate, nosql_id,=
 cf_to_update, value_to_incr=
ement)


 =A0File "/op=
t/acme/local/lib/python2.7/site-packages/s1audservice/statistics_service.py=
", line 397, in increment_counter


 =A0 =A0pycassa.ConsistencyLevel.ALL)

 =A0File "/op=
t/acme/local/lib/python2.7/site-packages/pycassa/columnfamily.py", line 1066, =
in add


 =A0 =A0allow_retries=3D<=
span style=3D"color:rgb(137,89,168)">self._allow_retries<=
span style=3D"color:rgb(77,77,76)">)


 =A0File "/op=
t/acme/local/lib/python2.7/site-packages/pycassa/pool.py", =
line 577, in execute


 =A0 =A0return getattr(conn, f)(*args, **kwargs)


 =A0File "/op=
t/acme/local/lib/python2.7/site-packages/pycassa/pool.py", =
line 148, in new_f


 =A0 =A0(self._retry_count, exc.__class__.__name__, exc))


MaximumRetryException: Retried 1 times. Last failure was EOFError<=
span style=3D"color:rgb(77,77,76)">:
I'm using=A0DataStax Enterprise version 3.2.2-1, with cassandra=A0= 1.2.12.2.
my application is written in python 2.7 and conne= cts cassandra through pycassa=A01.10.0, my thrift version is 0.9.1

here is the code that connects cassandra:

 __instances =3D {}
    __CONNECTION_POOL_SIZE_PER_HOST =3D 20

    def __init__(=
self, prefix<=
/span>):
        if not prefix:
            raise ValueError <=
span>("Invalid prefix for connection")
       =20
        self.=
ip_seeds =3D ConfigService.get().get_property(=
prefix + ".host")=
 + ":"\
                       + ConfigService.get().get_property(prefix + ".port")
        self.=
cluster_name =3D ConfigService.get().get_property(prefix + ".cluster_name")
        self.=
keyspace_name =3D ConfigService.get().get_property(prefix + ".keyspace")
        self.=
conec_pool_name =3D ConfigService.get().get_property=
(prefix + ".pool_name")
        self.=
con_pool =3D pycassa.Co=
nnectionPool(self.keyspace_name, server_li=
st=3D[=
self.ip_seeds], timeout=
=3D=
50)


this is the code that triggers the= exception:

cf_to_update.add(
             (counter_type_tag, long(str_date_for=
matted)),
             nosql_id,
             int(value),
             None,
             pycassa.ConsistencyLevel.ALL)


This same code is running successfully without any errors = with a single node cassandra cluster and very close configurations:
python 2.7 apache cassandra 1.2.8, pycassa 1.10.0, thrift 0.9.1=A0

any help would be greatly appreciated.



--
Ty= ler Hobbs
DataStax
 --001a11345a7a77105404f018f01a--