Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: AbstractCassandraDaemon.java (line 134) Exception in thread
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: 
 <CAJ7=NKsLoBcg=LRGbiv4w7CbYJtqjcaBTfO=ocn91odgpWe1aw@mail.gmail.com>
Date: Wed, 17 Jul 2013 21:59:36 +1200
Content-Transfer-Encoding: quoted-printable
Message-Id: <F3EE7856-D055-4C5E-B34C-30E5E6FFB124@thelastpickle.com>
References: 
 <CAJ7=NKsLoBcg=LRGbiv4w7CbYJtqjcaBTfO=ocn91odgpWe1aw@mail.gmail.com>
To: user@cassandra.apache.org

Double check the stack size is to set 100K see =
https://github.com/apache/cassandra/blob/cassandra-1.1/conf/cassandra-env.=
sh#L187

Cheers

-----------------
Aaron Morton
Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 17/07/2013, at 6:26 AM, Julio Quierati <julio.quierati@gmail.com> =
wrote:

> Hello,
>=20
> At least 2 times a day I'm having hundreds of log entry related to =
exception below, the network bottleneck seems, anyone know how to solve, =
or encountered this problem
>=20
> Using
>   C* version 1.1.4
>    and
> Java version "1.7.0_21"
> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
> AWS m1.xlarge
>=20
> ERROR [Thrift:967696] 2013-07-16 12:36:12,514 =
AbstractCassandraDaemon.java (line 134) Exception in thread =
Thread[Thrift:967696,5,main]
> java.lang.StackOverflowError
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:150)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:121)
> 	at =
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> 	at =
java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> 	at =
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> 	at =
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.jav=
a:127)
> 	at =
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> 	at =
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.ja=
va:129)
> 	at =
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:10=
1)
> 	at =
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> 	at =
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:37=
8)
> 	at =
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:29=
7)
> 	at =
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtoco=
l.java:204)
> 	at =
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
> 	at =
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(Cust=
omTThreadPoolServer.java:186)
> 	at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1145)
> 	at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:615)
> 	at java.lang.Thread.run(Thread.java:722)
> ERROR [Thrift:966717] 2013-07-16 12:36:17,152 =
AbstractCassandraDaemon.java (line 134) Exception in thread =
Thread[Thrift:966717,5,main]
> java.lang.StackOverflowError
> 	at java.net.SocketInputStream.socketRead0(Native Method)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:150)
> 	at java.net.SocketInputStream.read(SocketInputStream.java:121)
> 	at =
java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> 	at =
java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> 	at =
java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> 	at =
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.jav=
a:127)
> 	at =
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> 	at =
org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.ja=
va:129)
> 	at =
org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:10=
1)
> 	at =
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> 	at =
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:37=
8)
> 	at =
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:29=
7)
> 	at =
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtoco=
l.java:204)
> 	at =
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
> 	at =
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(Cust=
omTThreadPoolServer.java:186)
> 	at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1145)
> 	at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:615)
> 	at java.lang.Thread.run(Thread.java:722)
>  INFO [OptionalTasks:1] 2013-07-16 12:38:07,847 MeteredFlusher.java =
(line 62) flushing high-traffic column family CFS(Keyspace=3D'zm', =
ColumnFamily=3D'deviceConnectionLog') (estimated 200441950 bytes)
>  INFO [OptionalTasks:1] 2013-07-16 12:38:07,848 ColumnFamilyStore.java =
(line 659) Enqueuing flush of =
Memtable-deviceConnectionLog@64568169(20293642/200441950 serialized/live =
bytes, 114490 ops)
>  INFO [FlushWriter:4306] 2013-07-16 12:38:07,848 Memtable.java (line =
264) Writing Memtable-deviceConnectionLog@64568169(20293642/200441950 =
serialized/live bytes, 114490 ops)
>  INFO [FlushWriter:4306] 2013-07-16 12:38:09,154 Memtable.java (line =
305) Completed flushing =
/mnt/cassandra/data/zm/deviceConnectionLog/zm-deviceConnectionLog-he-5083-=
Data.db (5747695 bytes) for commitlog position =
ReplayPosition(segmentId=3D1282719983369784, position=3D28288612)
>  INFO [OptionalTasks:1] 2013-07-16 12:40:35,610 ColumnFamilyStore.java =
(line 659) Enqueuing flush of =
Memtable-userNotificationsHistory@1933122113(554/13847 serialized/live =
bytes, 10 ops)
>  INFO [FlushWriter:4307] 2013-07-16 12:40:35,611 Memtable.java (line =
264) Writing Memtable-userNotificationsHistory@1933122113(554/13847 =
serialized/live bytes, 10 ops)
>  INFO [OptionalTasks:1] 2013-07-16 12:40:35,611 ColumnFamilyStore.java =
(line 659) Enqueuing flush of =
Memtable-userPendingNotifications@1553457921(647/8729 serialized/live =
bytes, 10 ops)
>  INFO [FlushWriter:4307] 2013-07-16 12:40:35,717 Memtable.java (line =
305) Completed flushing =
/mnt/cassandra/data/zm/userNotificationsHistory/zm-userNotificationsHistor=
y-he-286-Data.db (488 bytes) for commitlog position =
ReplayPosition(segmentId=3D1283535713674915, position=3D261)
>=20
> PHP-FPM+NGINX
>=20
> [error] 1272#0: *2715913 FastCGI sent in stderr: "PHP message: Error =
performing describe_keyspace on nzmc-cass-01d.zm.com:9160: exception =
'Exception' with message 'describe_keyspace failed: unknown result' in =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thri=
ft/packages/cassandra/cassandra.Cassandra.client.php:1398
> Stack trace:
> #0 =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thri=
ft/packages/cassandra/cassandra.Cassandra.client.php(1347): =
cassandra\CassandraClient->recv_describe_keyspace()
> #1 [internal function]: =
cassandra\CassandraClient->describe_keyspace('zm')
> #2 =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpc=
assa/Connection/ConnectionPool.php(261): call_user_func_array(Array, =
Array)
> #3 =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpc=
assa/Connection/ConnectionPool.php(192): =
phpcassa\Connection\ConnectionPool->call('describe_keyspa...', 'zm')
> #4 =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/p...=

> PHP message: PHP Fatal error:  Uncaught exception 'Exception' with =
message 'describe_keyspace failed: unknown result' in =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thri=
ft/packages/cassandra/cassandra.Cassandra.client.php:1398
> Stack trace:
> #0 =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thri=
ft/packages/cassandra/cassandra.Cassandra.client.php(1347): =
cassandra\CassandraClient->recv_describe_keyspace()
> #1 [internal function]: =
cassandra\CassandraClient->describe_keyspace('zm')
> #2 =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpc=
assa/Connection/ConnectionPool.php(261): call_user_func_array(Array, =
Array)
> #3 =
/var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpc=
assa/Connection/ConnectionPool.php(192): =
phpcassa\Connection\ConnectionPool->call('describe_keyspa...', 'zm')
> #4 /var/www/zm-server/coreIncludes/data/cassandra
>=20
>=20
> Att,
>=20
> --
> Julio Quierati
> User Linux #492973
>=20
> OpenPGP Key: 0xC9A064FA578E0D60
> 8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60