Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of mmoores@real.com designates
 207.188.23.7 as permitted sender)
From: Michael Moores <mmoores@real.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wed, 20 Oct 2010 15:35:04 -0700
Subject: Re: Throttling ColumnFamilyRecordReader
Thread-Topic: Throttling ColumnFamilyRecordReader
Thread-Index: ActwpwrkojV/eSSsSae9ZnZ6oQQ5lQ==
Message-ID: <2902C786-A51B-4D86-A57A-51A98D38675C@real.com>
References: <E7C51F55-2E8E-4ACB-98C8-7AF5C1BDEA0B@real.com>
 <AANLkTi=N_=pji-CZZHMu96VMUZxcS28SFP76Thd6VL4o@mail.gmail.com>
In-Reply-To: <AANLkTi=N_=pji-CZZHMu96VMUZxcS28SFP76Thd6VL4o@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

sorry i had a misunderstanding of the MapRed report output.
i did reduce mapreduce.tasktracker.map.tasks.maximum (number of concurrent =
maps per node) from the default of 2 to 1.
i suppose if i want to do this on a per job/user basis i'll try out the had=
oop fair scheduler.


On Oct 19, 2010, at 1:27 PM, Jonathan Ellis wrote:

> (Moving to user@.)
>=20
> Isn't reducing the number of map tasks the easiest way to tune this?
>=20
> Also: in 0.7 you can use NetworkTopologyStrategy to designate a group
> of nodes as your hadoop "datacenter" so the workloads won't overlap.
>=20
> On Tue, Oct 19, 2010 at 3:22 PM, Michael Moores <mmoores@real.com> wrote:
>> Does it make sense to add some kind of throttle capability on the Column=
FamilyRecordReader for Hadoop?
>>=20
>> If I have 60 or so Map tasks running at the same time when the cluster i=
s already heavily loaded with OLTP operations, I can get some decreased on-=
line performance
>> that may not be acceptable.  (I'm loading an 8 node cluster with 2000 TP=
S.)  By default my cluster of 8 nodes (which are also the Hadoop JobTracker=
 nodes) has 8 Map tasks per node making the get_range_slices call, based on=
 what the ColumnFamilyInputFormat has calculated from my token ranges.
>> I can increase the inputSplitSize  (ConfigHelper.setInputSplitSIze()) so=
 that there
>> is only one Map task per node, and this helps quite a bit.
>>=20
>> But is it reasonable to provide a configurable sleep to cause a wait in =
between smaller size range queries?  That would stretch out the Map time
>> and let the OLTP processing be less affected.
>>=20
>>=20
>> --Michael
>>=20
>>=20
>>=20
>=20
>=20
>=20
> --=20
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com