Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Message-ID: <14386873.73611274039562002.JavaMail.jira@thor>
Date: Sun, 16 May 2010 15:52:42 -0400 (EDT)
From: "Joost Ouwerkerk (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Subject: [jira] Updated: (CASSANDRA-1096) Sequential splits causing
 unbalanced MapReduce load
In-Reply-To: <16001440.73591274039442452.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/CASSANDRA-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joost Ouwerkerk updated CASSANDRA-1096:
---------------------------------------

    Attachment: CASSANDRA-1096.patch

I added Collections.shuffle(splits) before returning the splits in getSplits().  As a result, the load is much better distributed, throughput  was increased (about 3X in my case) and TimedOutExceptions were all but eliminated.

> Sequential splits causing unbalanced MapReduce load
> ---------------------------------------------------
>
>                 Key: CASSANDRA-1096
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1096
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>    Affects Versions: 0.6
>            Reporter: Joost Ouwerkerk
>         Attachments: CASSANDRA-1096.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Since CassandraInputFormat returns an ordered list of splits, when there are many splits (e.g. hundreds or more) the load on cassandra is horribly unbalanced.  e.g. if I have 30 tasks processing 600 splits, then the rows for the first 30 splits are all located on the same one or two nodes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.