Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of filippo@ntoklo.com designates
 74.125.82.44 as permitted sender)
Date: Tue, 1 May 2012 16:58:14 +0100
From: Filippo Diotalevi <filippo@ntoklo.com>
To: user@cassandra.apache.org
Message-ID: <FE06BA85376D4B5EBE7374981B5F0483@ntoklo.com>
Subject: How Cassandra determines the splits
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="4fa00816_fa085b0_166"

--4fa00816_fa085b0_166
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hi,
I'm having problems in my Cassandra/Hadoop (1.0.8 + cdh3u3) cluster related to how cassandra splits the data to be processed by Hadoop.

I'm currently testing a map reduce job, starting from a CF of roughly 1500 rows, with 

cassandra.input.split.size 10
cassandra.range.batch.size 1

but what I consistently see is that, while most of the task have 1-20 rows assigned each, one of them is assigned 400+ rows, which gives me all sort of problems in terms of timeouts and memory consumption (not to mention seeing the mapper progress bar going to 4000% and more).

Do you have any suggestion to solve/troublehsoot this issue?

-- 
Filippo Diotalevi

--4fa00816_fa085b0_166
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

<div>Hi,</div><div>I'm having problems in my Cassandra/Hadoop (1.0.8 + cd=
h3u3) cluster related to how cassandra splits the data to be processed by=
 Hadoop.</div><div><br></div><div>I'm currently testing a map reduce job,=
 starting from a C=46 of roughly 1500 rows, with&nbsp;</div><div><br></di=
v><div>cassandra.input.split.size 10</div><div>cassandra.range.batch.size=
 1</div><div><br></div><div>but what I consistently see is that, while mo=
st of the task have 1-20 rows assigned each, one of them is assigned 400+=
 rows, which gives me all sort of problems in terms of timeouts and memor=
y consumption (not to mention seeing the mapper progress bar going to 400=
0% and more).</div><div><br></div><div>Do you have any suggestion to solv=
e/troublehsoot this issue=3F</div><div><br></div><div>--&nbsp;</div><div>=
=46ilippo Diotalevi</div>
--4fa00816_fa085b0_166--