Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.82.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=nyUIWSPvHT6rIak1tb4gNJsK4LfZpaLVPBFf5gV6/+aiznbveGf9xrfSgaTgW5PV+1
         WoGdobswa2dF3djxOL+U0LI3HEW/3PVfJxhHyBVfZteBFXuMdudk4+wAfzMMcYzxzNx+
         IrnJz9VWGVM9D98PW+3Cocp3wU2pM/7GA0kL4=
MIME-Version: 1.0
In-Reply-To: <x2y775e31411004230739m6f87e1e7vdadfe95603265a14@mail.gmail.com>
References: <g2z775e31411004221927w774b958bta8432c889a932f42@mail.gmail.com>
	<u2we06563881004222009u995a3d8az649d75088e2a2f8b@mail.gmail.com>
	<20DF20F5-F894-40B3-AFB8-70D18F61E2A9@oskarsson.nu>
 <y2je06563881004230639i7af7217eoc70f5ba7f0d9e892@mail.gmail.com>
	<x2y775e31411004230739m6f87e1e7vdadfe95603265a14@mail.gmail.com>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Fri, 23 Apr 2010 09:47:15 -0500
Message-ID: <s2se06563881004230747kad9694bcm5b61ed2657eee493@mail.gmail.com>
Subject: Re: MapReduce, Timeouts and Range Batch Size
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You could look into it, but it's not going to be an easy backport
since SSTableReader and SSTableScanner got split into two classes in
trunk.

On Fri, Apr 23, 2010 at 9:39 AM, Joost Ouwerkerk <joost@openplaces.org> wro=
te:
> Awesome. =A0In the meantime, I hacked something similar myself. =A0The
> performance difference does not appear to be material. =A0I think the rea=
l
> killer is the get_range_slices call. =A0Relative to that, the cost of get=
ting
> the connection appears to be more or less trivial. =A0What can I do to
> alleviate that cost? =A0CASSANDRA-821 looks interesting -- can I apply th=
at to
> 0.6.1 ?
> joost.
> On Fri, Apr 23, 2010 at 9:39 AM, Jonathan Ellis <jbellis@gmail.com> wrote=
:
>>
>> Great! =A0Created https://issues.apache.org/jira/browse/CASSANDRA-1017
>> to track this.
>>
>> On Fri, Apr 23, 2010 at 4:12 AM, Johan Oskarsson <johan@oskarsson.nu>
>> wrote:
>> > I have written some code to avoid thrift reconnection, it just keeps t=
he
>> > connection open between get_range_slices calls.
>> > I can extract that and put it up but not until early next week.
>> >
>> > /Johan
>> >
>> > On 23 apr 2010, at 05.09, Jonathan Ellis wrote:
>> >
>> >> That would be an easy win, sure.
>> >>
>> >> On Thu, Apr 22, 2010 at 9:27 PM, Joost Ouwerkerk <joost@openplaces.or=
g>
>> >> wrote:
>> >>> I was getting client timeouts in ColumnFamilyRecordReader.maybeInit(=
)
>> >>> when
>> >>> MapReducing. =A0So I've reduced the Range Batch Size to 256 (from 40=
96)
>> >>> and
>> >>> this seems to have fixed my problem, although it has slowed things
>> >>> down a
>> >>> bit -- presumably because there are 16x more calls to
>> >>> get_range_slices.
>> >>> While I was in that code I noticed that a new client was being creat=
ed
>> >>> for
>> >>> each batch get. =A0By decreasing the batch size, I've increased this
>> >>> overhead. =A0I'm thinking of re-writing ColumnFamilyRecordReader to =
do
>> >>> some
>> >>> connection pooling. =A0Anyone have any thoughts on that?
>> >>> joost.
>> >>>
>> >
>> >
>
>