cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Chia (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-6754) communication exceptions between nodes
Date Sat, 22 Feb 2014 02:04:19 GMT
David Chia created CASSANDRA-6754:
-------------------------------------

             Summary: communication exceptions between nodes
                 Key: CASSANDRA-6754
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6754
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.3 LTS
Release:	12.04
Codename:	precise

AWS: i2.xlarge

{code}
INFO  22:34:40 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0
INFO  22:34:40 Heap size: 12777553920/12777553920
INFO  22:34:40 Code Cache Non-heap memory: init = 2555904(2496K) used = 621632(607K) committed
= 2555904(2496K) max = 50331648(49152K)
INFO  22:34:40 Par Eden Space Heap memory: init = 859045888(838912K) used = 137447616(134226K)
committed = 859045888(838912K) max = 859045888(838912K)
INFO  22:34:40 Par Survivor Space Heap memory: init = 107347968(104832K) used = 0(0K) committed
= 107347968(104832K) max = 107347968(104832K)
INFO  22:34:40 CMS Old Gen Heap memory: init = 11811160064(11534336K) used = 1433816(1400K)
committed = 11811160064(11534336K) max = 11811160064(11534336K)
INFO  22:34:40 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 18654512(18217K)
committed = 21757952(21248K) max = 85983232(83968K)
INFO  22:34:40 Classpath: /usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.6.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.0.13.jar:/usr/share/cassandra/lib/logback-core-1.0.13.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar:/usr/share/cassandra/CustomAgent.jar:/usr/share/cassandra/apache-cassandra-2.1.0~beta1.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.0~beta1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/jna.jar:/usr/share/cassandra/mx4j-tools.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.6.jar:/usr/share/cassandra/CustomAgent.jar:/usr/local/jcollectd/jcollectd.jar
{code}

{code:title=Node configuration}
[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batchlog_replay_throttle_in_kb=1024;
cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=sketchy_staging_test;
column_index_size_in_kb=64; commitlog_directory=/mnt/cassandra/commitlog; commitlog_segment_size_in_mb=32;
commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_preheat_key_cache=true;
compaction_throughput_mb_per_sec=64; concurrent_counter_writes=32; concurrent_reads=128; concurrent_writes=128;
counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000;
cross_node_timeout=false; data_file_directories=[/mnt/cassandra/data]; disk_failure_policy=stop;
dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100;
endpoint_snitch=SimpleSnitch; flush_directory=/mnt/cassandra/flush; hinted_handoff_enabled=true;
hinted_handoff_throttle_in_kb=1024; in_memory_compaction_limit_in_mb=64; incremental_backups=false;
index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false;
internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=1024; listen_address=10.9.163.158;
max_hint_window_in_ms=14400000; max_hints_delivery_threads=2; memtable_cleanup_threshold=0.4;
memtable_total_space_in_mb=2048; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner;
permissions_validity_in_ms=2000; preheat_kernel_page_cache=false; range_request_timeout_in_ms=10000;
read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler;
request_timeout_in_ms=10000; row_cache_save_period=14400; row_cache_size_in_mb=1024; rpc_address=0.0.0.0;
rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/mnt/cassandra/cache;
seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=10.71.141.38,10.218.142.35}]}];
server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001;
start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15;
tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=true; trickle_fsync_interval_in_kb=10240;
truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
{code}

            Reporter: David Chia
             Fix For: 2.1 beta1


After upgrading from 2.0.5 to 2.1.0 beta1, I get "a lot" (most of them are repeated) of exceptions.
They may be related, so I encapsulate all of the exceptions in this ticket.

{code:title=Exception 1}
ERROR [HintedHandoff:1] 2014-02-22 01:44:59,509 - Scanned over 100000 tombstones in system.hints;
query aborted (see tombstone_fail_threshold)
ERROR [HintedHandoff:1] 2014-02-22 01:44:59,514 - Exception in thread Thread[HintedHandoff:1,1,main]
org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
        at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:213)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:123)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:73)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:283)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:51)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1846)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1639)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:375)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:332)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:94)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:558)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0]
        at java.lang.Thread.run(Unknown Source) ~[na:1.7.0]
{code}

{code:title=Exception 2}
ERROR [WRITE-/10.9.163.158] 2014-02-22 01:46:06,279 - error writing to /10.9.163.158
java.lang.NullPointerException: null
        at org.apache.cassandra.db.ArrayBackedSortedColumns.sortCells(ArrayBackedSortedColumns.java:116)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.maybeSortCells(ArrayBackedSortedColumns.java:103)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.getColumnCount(ArrayBackedSortedColumns.java:313)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilySerializer.contentSerializedSize(ColumnFamilySerializer.java:117)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilySerializer.serializedSize(ColumnFamilySerializer.java:132)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:337)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:263)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:116) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:273)
[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:225)
[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:163)
[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
INFO  [HANDSHAKE-/10.9.163.158] 2014-02-22 01:46:06,286 - Handshaking version with /10.9.163.158
ERROR [MutationStage:107] 2014-02-22 01:46:06,682 - Exception in thread Thread[MutationStage:107,5,main]
java.lang.RuntimeException: java.nio.BufferOverflowException
        at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2006)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0]
        at java.lang.Thread.run(Unknown Source) ~[na:1.7.0]
Caused by: java.nio.BufferOverflowException: null
        at java.nio.Buffer.nextPutIndex(Unknown Source) ~[na:1.7.0]
        at java.nio.DirectByteBuffer.put(Unknown Source) ~[na:1.7.0]
        at org.apache.cassandra.io.util.ByteBufferOutputStream.write(ByteBufferOutputStream.java:38)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.io.util.ChecksummedOutputStream.write(ChecksummedOutputStream.java:46)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at java.io.DataOutputStream.writeInt(Unknown Source) ~[na:1.7.0]
        at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:77) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilySerializer.serialize(ColumnFamilySerializer.java:70)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Mutation$MutationSerializer.serialize(Mutation.java:277)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:222) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:193) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:349) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:328) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:205) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:940)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2002)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        ... 3 common frames omitted
{code}

{code:title=Exception 3}
ERROR [WRITE-/10.218.142.35] 2014-02-22 01:46:08,670 - error writing to /10.218.142.35
java.lang.IllegalArgumentException: fromIndex(3) > toIndex(2)
        at java.util.TimSort.rangeCheck(Unknown Source) ~[na:1.7.0]
        at java.util.TimSort.sort(Unknown Source) ~[na:1.7.0]
        at java.util.Arrays.sort(Unknown Source) ~[na:1.7.0]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.sortCells(ArrayBackedSortedColumns.java:113)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.maybeSortCells(ArrayBackedSortedColumns.java:103)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.getSortedColumns(ArrayBackedSortedColumns.java:301)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamily.iterator(ColumnFamily.java:445) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilySerializer.contentSerializedSize(ColumnFamilySerializer.java:119)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilySerializer.serializedSize(ColumnFamilySerializer.java:132)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:337)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:263)
~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:116) ~[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:273)
[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:225)
[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
        at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:163)
[apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT]
{code}

Note: apache-cassandra-2.1.0~beta1.jar:2.1.0-beta1-SNAPSHOT is to fix the issue from [CASSANDRA-6753|https://issues.apache.org/jira/browse/CASSANDRA-6753?focusedCommentId=13909094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13909094]

Let me know if there is any other info I should provide. :D



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message