cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Rolf (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-6665) Batching in CqlRecordWriter
Date Thu, 06 Feb 2014 13:44:10 GMT
Christian Rolf created CASSANDRA-6665:
-----------------------------------------

             Summary: Batching in CqlRecordWriter
                 Key: CASSANDRA-6665
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6665
             Project: Cassandra
          Issue Type: Improvement
          Components: Hadoop
         Environment: Cluster of 12 nodes, each node with 256-384 vnodes. RPC threads capped
at 2048.
            Reporter: Christian Rolf
            Priority: Minor


We're writing from Pig map tasks, about 20 million records of one integer each. 
For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000 threads per mapper.
This obviously overloads the nodes, since the number of RPC threads are capped, and the write
fails. 
Also, each transfer is only in the order of a few bytes of payload. Clearly batching is a
good solution.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message