Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 7 Feb 2014 09:47:19 +0000 (UTC)
From: "Christian Rolf (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12693559.1391694201102.3381.1391766439105@arcas>
In-Reply-To: <JIRA.12693559.1391694201102@arcas>
References: <JIRA.12693559.1391694201102@arcas>
Subject: [jira] [Commented] (CASSANDRA-6665) Batching in CqlRecordWriter
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-6665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13894348#comment-13894348 ] 

Christian Rolf commented on CASSANDRA-6665:
-------------------------------------------

Thanks for the quick feedback John! 
The biggest issue we've seen is on larger rings, where the write tasks quickly run out of heap (caching is per replica). That's why I left the batching disabled by default.

> Batching in CqlRecordWriter
> ---------------------------
>
>                 Key: CASSANDRA-6665
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6665
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Hadoop
>         Environment: Cluster of 12 nodes, each node with 256-384 vnodes. RPC threads capped at 2048.
>            Reporter: Christian Rolf
>            Priority: Minor
>         Attachments: batchWrite.txt
>
>
> We're writing from Pig map tasks, about 20 million records of one integer each. 
> For the case of 12 nodes, with 256-384 vnodes per node, we get around 4000 threads per mapper. This obviously overloads the nodes, since the number of RPC threads are capped, and the write fails. 
> Also, each transfer is only in the order of a few bytes of payload. Clearly batching is a good solution.


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)