cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sharaf Ali <sharafjaf...@live.com>
Subject Insert failed after some time in cassandra with timeout‏
Date Fri, 30 May 2014 10:38:58 GMT
I have installed Cassandra 2.0 On CentOS6.5 Server and and while testing
 simple records everything is working fine, Now I have to upload 600 
billion rows, when I use COPY on cqlsh it failed after 5 minutes and 
approx rows inserted are 0.2 million with rpc timeout, then I opted for 
pycasso and parsed csv and tried to import using inserts commands, after
 every 10K records, We opted to close connection and develop new 
connection again. but after around 60k records it failed with with 
timeout.

My debug trace shows something this while server is not accepting inserts, without any activity
it's still busy.

   
 DEBUG [OptionalTasks:1] 2014-05-30 04:34:16,305 MeteredFlusher.java 
(line 41) Currently flushing 269227480 bytes of 2047868928 max
    
DEBUG [OptionalTasks:1] 2014-05-30 04:34:17,306 MeteredFlusher.java 
(line 41) Currently flushing 269227480 bytes of 2047868928 max
    
DEBUG [OptionalTasks:1] 2014-05-30 04:34:18,306 MeteredFlusher.java 
(line 41) Currently flushing 269227480 bytes of 2047868928 max
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy
for schema_triggers is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy
for compaction_history is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy
for batchlog is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy
for sstable_activity is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java (line 298) retryPolicy
for peer_events is 0.99
   
 DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,012 ColumnFamilyStore.java 
(line 298) retryPolicy for compactions_in_progress is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy
for hints is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy
for schema_keyspaces is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy
for range_xfers is 0.99
   
 DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java 
(line 298) retryPolicy for schema_columnfamilies is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy
for NodeIdInfo is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy
for paxos is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,013 ColumnFamilyStore.java (line 298) retryPolicy
for schema_columns is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy
for IndexInfo is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy
for peers is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,014 ColumnFamilyStore.java (line 298) retryPolicy
for local is 0.99
   
 DEBUG [OptionalTasks:1] 2014-05-30 04:34:19,307 MeteredFlusher.java 
(line 41) Currently flushing 269227480 bytes of 2047868928 max
    
DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,307 MeteredFlusher.java 
(line 41) Currently flushing 269227480 bytes of 2047868928 max
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy
for backup_calls is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy
for sessions is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:20,716 ColumnFamilyStore.java (line 298) retryPolicy
for events is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:21,308 MeteredFlusher.java (line 41) Currently
flushing 269

while When I try to insert records it show error like this in debug log.

    DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,717 ColumnFamilyStore.java (line 298) retryPolicy
for backup_calls is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,717 ColumnFamilyStore.java (line 298) retryPolicy
for sessions is 0.99
    DEBUG [OptionalTasks:1] 2014-05-30 04:34:40,718 ColumnFamilyStore.java (line 298) retryPolicy
for events is 0.99
   
 DEBUG [Thrift:24] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java 
(line 211) Thrift transport error occurred during processing of message.
    org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
   
 DEBUG [Thrift:19] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java 
(line 211) Thrift transport error occurred during processing of message.
    org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
   
 DEBUG [Thrift:21] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java 
(line 211) Thrift transport error occurred during processing of message.
    org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
   
 DEBUG [Thrift:1] 2014-05-30 04:34:40,775 CustomTThreadPoolServer.java 
(line 211) Thrift transport error occurred during processing of message.
    org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
        at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
    DEBUG [Thrift:21] 2014-05-30 04:34:40,789 CassandraServer.java (line 949) batch_mutate
    DEBUG [Thrift:19] 2014-05-30 04:34:41,314 CassandraServer.java (line 949) batch_mutate
   
 DEBUG [OptionalTasks:1] 2014-05-30 04:34:41,317 MeteredFlusher.java 
(line 41) Currently flushing 269227480 bytes of 2047868928 max
    DEBUG [Thrift:24] 2014-05-30 04:34:41,858 CassandraServer.java (line 949) batch_mutate


Any Idea about what is happening?, after insertion failed it will not work until server is
restarted.


**Node Details:**

There is only a single node and details are given below,

    # nodetool describecluster
    Cluster Information:
        Name: Test Cluster
        Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
        Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
        Schema versions:
            93403140-5bd6-3cfa-9973-b18df44227f9: [127.0.0.1]

**My Node is up and running?**

    # nodetool status
    Note: Ownership information does not include topology; for complete information, specify
a keyspace
    Datacenter: datacenter1
    =======================
    Status=Up/Down
    |/ State=Normal/Leaving/Joining/Moving
    --  Address    Load       Owns   Host ID                               Token         
                          Rack
    UN  127.0.0.1  35.05 MB   100.0%  6bc340b2-4534-400f-b23d-9fe3ed0c6971  -9120530846114649324
                    rack1

**Server Details:**

    OS CentOS release 6.5 (Final) 
    hdd 8Tb all SSD Drives 
    RAM 32GB 
    Proccessor: model name : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 24 Core(2 physical


processors with 12 cores each) 		 	   		  
Mime
View raw message