incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julio Quierati <julio.quier...@gmail.com>
Subject Re: AbstractCassandraDaemon.java (line 134) Exception in thread
Date Wed, 17 Jul 2013 16:05:38 GMT
System up again xD

nodetool removetoken 129020446491903175361975561488312102412
nodetool removetoken force
nodetool removetoken 42514072861213193928274919594412671125
nodetool removetoken force
nodetool removetoken 99246027852756676471313599864858497886
nodetool removetoken force




--
Julio Quierati
User Linux #492973

OpenPGP Key: 0xC9A064FA578E0D60
8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60


2013/7/17 Julio Quierati <julio.quierati@gmail.com>

> Hi Aaron,
>
> My cassandra-env is by setting "-Xss160k" changed to "-Xss100k".
>
> if [ "`uname`" = "Linux" ] ; then
>     # reduce the per-thread stack size to minimize the impact of Thrift
>     # thread-per-client.  (Best practice is for client connections to
>     # be pooled anyway.) Only do so on Linux where it is known to be
>     # supported.
>     if startswith "$JVM_VERSION" '1.7.'
>     then
>         JVM_OPTS="$JVM_OPTS -Xss160k"
>     else
>         JVM_OPTS="$JVM_OPTS -Xss128k"
>     fi
> fi
>
> cassandra error after restart.
>
> The stack size specified is too small, Specify at least 160k
> Cannot create Java VM
> Service exit with a return value of 1
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-4275
> I set 256k.
>
> A new problem, after restart, the nodetool ring, this viewing old nodes
> removed a 1 months ago, these nodesnot exist anymore :<
>
>
> nodetool ring
> Note: Ownership information does not include topology, please specify a
> keyspace.
> Address         DC          Rack        Status State   Load
>  Owns                Token
>
>                  134691819273918816419698471612174905938
> 172.16.1.48     datacenter1 rack1       Down   Normal  ?
> 45.82%              42514072861213193928274919594412671125
> 172.16.4.212    datacenter1 rack1       Up     Normal  21.59 GB
>  4.18%               49621227543684200553854819754232853074
> 172.16.3.221    datacenter1 rack1       Down   Normal  ?
> 29.17%              99246027852756676471313599864858497886
> 172.16.5.57     datacenter1 rack1       Down   Normal  ?
> 17.50%              129020446491903175361975561488312102412
> 172.16.3.93     datacenter1 rack1       Up     Normal  21.96 GB
>  3.33%               134691819273918816419698471612174905938
>
>
> I have 2 nodes, now errors, system is down now :<
> 172.16.3.93      datacenter1 rack1       Up     Normal  21.96 GB
>  3.33%               134691819273918816419698471612174905938
> 172.16.4.212    datacenter1 rack1       Up     Normal  21.59 GB
>  4.18%               49621227543684200553854819754232853074
>
>
>
> --
> Julio Quierati
> User Linux #492973
>
> OpenPGP Key: 0xC9A064FA578E0D60
> 8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60
>
>
> 2013/7/17 aaron morton <aaron@thelastpickle.com>
>
>> Double check the stack size is to set 100K see
>> https://github.com/apache/cassandra/blob/cassandra-1.1/conf/cassandra-env.sh#L187
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Cassandra Consultant
>> New Zealand
>>
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 17/07/2013, at 6:26 AM, Julio Quierati <julio.quierati@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > At least 2 times a day I'm having hundreds of log entry related to
>> exception below, the network bottleneck seems, anyone know how to solve, or
>> encountered this problem
>> >
>> > Using
>> >   C* version 1.1.4
>> >    and
>> > Java version "1.7.0_21"
>> > Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
>> > Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
>> > AWS m1.xlarge
>> >
>> > ERROR [Thrift:967696] 2013-07-16 12:36:12,514
>> AbstractCassandraDaemon.java (line 134) Exception in thread
>> Thread[Thrift:967696,5,main]
>> > java.lang.StackOverflowError
>> >       at java.net.SocketInputStream.socketRead0(Native Method)
>> >       at java.net.SocketInputStream.read(SocketInputStream.java:150)
>> >       at java.net.SocketInputStream.read(SocketInputStream.java:121)
>> >       at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> >       at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>> >       at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> >       at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>> >       at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> >       at
>> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>> >       at
>> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>> >       at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> >       at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>> >       at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>> >       at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>> >       at
>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
>> >       at
>> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>> >       at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >       at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >       at java.lang.Thread.run(Thread.java:722)
>> > ERROR [Thrift:966717] 2013-07-16 12:36:17,152
>> AbstractCassandraDaemon.java (line 134) Exception in thread
>> Thread[Thrift:966717,5,main]
>> > java.lang.StackOverflowError
>> >       at java.net.SocketInputStream.socketRead0(Native Method)
>> >       at java.net.SocketInputStream.read(SocketInputStream.java:150)
>> >       at java.net.SocketInputStream.read(SocketInputStream.java:121)
>> >       at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>> >       at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>> >       at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>> >       at
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>> >       at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> >       at
>> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>> >       at
>> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
>> >       at
>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>> >       at
>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>> >       at
>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>> >       at
>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>> >       at
>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
>> >       at
>> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
>> >       at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >       at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >       at java.lang.Thread.run(Thread.java:722)
>> >  INFO [OptionalTasks:1] 2013-07-16 12:38:07,847 MeteredFlusher.java
>> (line 62) flushing high-traffic column family CFS(Keyspace='zm',
>> ColumnFamily='deviceConnectionLog') (estimated 200441950 bytes)
>> >  INFO [OptionalTasks:1] 2013-07-16 12:38:07,848 ColumnFamilyStore.java
>> (line 659) Enqueuing flush of Memtable-deviceConnectionLog@64568169(20293642/200441950
>> serialized/live bytes, 114490 ops)
>> >  INFO [FlushWriter:4306] 2013-07-16 12:38:07,848 Memtable.java (line
>> 264) Writing Memtable-deviceConnectionLog@64568169(20293642/200441950
>> serialized/live bytes, 114490 ops)
>> >  INFO [FlushWriter:4306] 2013-07-16 12:38:09,154 Memtable.java (line
>> 305) Completed flushing
>> /mnt/cassandra/data/zm/deviceConnectionLog/zm-deviceConnectionLog-he-5083-Data.db
>> (5747695 bytes) for commitlog position
>> ReplayPosition(segmentId=1282719983369784, position=28288612)
>> >  INFO [OptionalTasks:1] 2013-07-16 12:40:35,610 ColumnFamilyStore.java
>> (line 659) Enqueuing flush of Memtable-userNotificationsHistory@1933122113(554/13847
>> serialized/live bytes, 10 ops)
>> >  INFO [FlushWriter:4307] 2013-07-16 12:40:35,611 Memtable.java (line
>> 264) Writing Memtable-userNotificationsHistory@1933122113(554/13847
>> serialized/live bytes, 10 ops)
>> >  INFO [OptionalTasks:1] 2013-07-16 12:40:35,611 ColumnFamilyStore.java
>> (line 659) Enqueuing flush of Memtable-userPendingNotifications@
>> 1553457921(647/8729 serialized/live bytes, 10 ops)
>> >  INFO [FlushWriter:4307] 2013-07-16 12:40:35,717 Memtable.java (line
>> 305) Completed flushing
>> /mnt/cassandra/data/zm/userNotificationsHistory/zm-userNotificationsHistory-he-286-Data.db
>> (488 bytes) for commitlog position
>> ReplayPosition(segmentId=1283535713674915, position=261)
>> >
>> > PHP-FPM+NGINX
>> >
>> > [error] 1272#0: *2715913 FastCGI sent in stderr: "PHP message: Error
>> performing describe_keyspace on nzmc-cass-01d.zm.com:9160: exception
>> 'Exception' with message 'describe_keyspace failed: unknown result' in
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thrift/packages/cassandra/cassandra.Cassandra.client.php:1398
>> > Stack trace:
>> > #0
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thrift/packages/cassandra/cassandra.Cassandra.client.php(1347):
>> cassandra\CassandraClient->recv_describe_keyspace()
>> > #1 [internal function]:
>> cassandra\CassandraClient->describe_keyspace('zm')
>> > #2
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpcassa/Connection/ConnectionPool.php(261):
>> call_user_func_array(Array, Array)
>> > #3
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpcassa/Connection/ConnectionPool.php(192):
>> phpcassa\Connection\ConnectionPool->call('describe_keyspa...', 'zm')
>> > #4
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/p...
>> > PHP message: PHP Fatal error:  Uncaught exception 'Exception' with
>> message 'describe_keyspace failed: unknown result' in
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thrift/packages/cassandra/cassandra.Cassandra.client.php:1398
>> > Stack trace:
>> > #0
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/thrift/packages/cassandra/cassandra.Cassandra.client.php(1347):
>> cassandra\CassandraClient->recv_describe_keyspace()
>> > #1 [internal function]:
>> cassandra\CassandraClient->describe_keyspace('zm')
>> > #2
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpcassa/Connection/ConnectionPool.php(261):
>> call_user_func_array(Array, Array)
>> > #3
>> /var/www/zm-server/coreIncludes/data/cassandra-odm/noSql/phpcassa-lib/phpcassa/Connection/ConnectionPool.php(192):
>> phpcassa\Connection\ConnectionPool->call('describe_keyspa...', 'zm')
>> > #4 /var/www/zm-server/coreIncludes/data/cassandra
>> >
>> >
>> > Att,
>> >
>> > --
>> > Julio Quierati
>> > User Linux #492973
>> >
>> > OpenPGP Key: 0xC9A064FA578E0D60
>> > 8D70 B111 ECE9 D3E9 E661 305B C9A0 64FA 578E 0D60
>>
>>
>

Mime
View raw message