cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Christian Rolf (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-6665) Batching in CqlRecordWriter
Date Fri, 07 Feb 2014 09:47:19 GMT


Christian Rolf commented on CASSANDRA-6665:

Thanks for the quick feedback John! 
The biggest issue we've seen is on larger rings, where the write tasks quickly run out of
heap (caching is per replica). That's why I left the batching disabled by default.

> Batching in CqlRecordWriter
> ---------------------------
>                 Key: CASSANDRA-6665
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Hadoop
>         Environment: Cluster of 12 nodes, each node with 256-384 vnodes. RPC threads
capped at 2048.
>            Reporter: Christian Rolf
>            Priority: Minor
>         Attachments: batchWrite.txt
> We're writing from Pig map tasks, about 20 million records of one integer each. 
> For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000 threads per
mapper. This obviously overloads the nodes, since the number of RPC threads are capped, and
the write fails. 
> Also, each transfer is only in the order of a few bytes of payload. Clearly batching
is a good solution.

This message was sent by Atlassian JIRA

View raw message