Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 87810 invoked from network); 23 Apr 2010 14:40:17 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 23 Apr 2010 14:40:17 -0000 Received: (qmail 16103 invoked by uid 500); 23 Apr 2010 14:40:16 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 16053 invoked by uid 500); 23 Apr 2010 14:40:16 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 16045 invoked by uid 99); 23 Apr 2010 14:40:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 14:40:16 +0000 X-ASF-Spam-Status: No, hits=1.6 required=10.0 tests=AWL,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.212.44] (HELO mail-vw0-f44.google.com) (209.85.212.44) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 23 Apr 2010 14:40:11 +0000 Received: by vws1 with SMTP id 1so818209vws.31 for ; Fri, 23 Apr 2010 07:39:47 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.73.135 with SMTP id q7mr158004qcj.41.1272033585050; Fri, 23 Apr 2010 07:39:45 -0700 (PDT) Received: by 10.229.189.130 with HTTP; Fri, 23 Apr 2010 07:39:45 -0700 (PDT) In-Reply-To: References: <20DF20F5-F894-40B3-AFB8-70D18F61E2A9@oskarsson.nu> Date: Fri, 23 Apr 2010 10:39:45 -0400 Message-ID: Subject: Re: MapReduce, Timeouts and Range Batch Size From: Joost Ouwerkerk To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016364ee0ecfabf5b0484e86542 --0016364ee0ecfabf5b0484e86542 Content-Type: text/plain; charset=ISO-8859-1 Awesome. In the meantime, I hacked something similar myself. The performance difference does not appear to be material. I think the real killer is the get_range_slices call. Relative to that, the cost of getting the connection appears to be more or less trivial. What can I do to alleviate that cost? CASSANDRA-821 looks interesting -- can I apply that to 0.6.1 ? joost. On Fri, Apr 23, 2010 at 9:39 AM, Jonathan Ellis wrote: > Great! Created https://issues.apache.org/jira/browse/CASSANDRA-1017 > to track this. > > On Fri, Apr 23, 2010 at 4:12 AM, Johan Oskarsson > wrote: > > I have written some code to avoid thrift reconnection, it just keeps the > connection open between get_range_slices calls. > > I can extract that and put it up but not until early next week. > > > > /Johan > > > > On 23 apr 2010, at 05.09, Jonathan Ellis wrote: > > > >> That would be an easy win, sure. > >> > >> On Thu, Apr 22, 2010 at 9:27 PM, Joost Ouwerkerk > wrote: > >>> I was getting client timeouts in ColumnFamilyRecordReader.maybeInit() > when > >>> MapReducing. So I've reduced the Range Batch Size to 256 (from 4096) > and > >>> this seems to have fixed my problem, although it has slowed things down > a > >>> bit -- presumably because there are 16x more calls to get_range_slices. > >>> While I was in that code I noticed that a new client was being created > for > >>> each batch get. By decreasing the batch size, I've increased this > >>> overhead. I'm thinking of re-writing ColumnFamilyRecordReader to do > some > >>> connection pooling. Anyone have any thoughts on that? > >>> joost. > >>> > > > > > --0016364ee0ecfabf5b0484e86542 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Awesome. =A0In the meantime, I hacked something similar myself. =A0The perf= ormance difference does not appear to be material. =A0I think the real kill= er is the get_range_slices call. =A0Relative to that, the cost of getting t= he connection appears to be more or less trivial. =A0What can I do to allev= iate that cost? =A0CASSANDRA-821 looks interesting -- can I apply that to 0= .6.1 ?
joost.

On Fri, Apr 23, 2010 at 9:3= 9 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
Great! =A0Created https://issues.apache.org/jira/browse/CASSANDRA-1= 017
to track this.

On Fri, Apr 23, 2010 at 4:12 AM, Johan Oskarsson <johan@oskarsson.nu> wrote:
> I have written some code to avoid thrift reconnection, it just keeps t= he connection open between get_range_slices calls.
> I can extract that and put it up but not until early next week.
>
> /Johan
>
> On 23 apr 2010, at 05.09, Jonathan Ellis wrote:
>
>> That would be an easy win, sure.
>>
>> On Thu, Apr 22, 2010 at 9:27 PM, Joost Ouwerkerk <joost@openplaces.org> wrote:
>>> I was getting client timeouts in ColumnFamilyRecordReader.mayb= eInit() when
>>> MapReducing. =A0So I've reduced the Range Batch Size to 25= 6 (from 4096) and
>>> this seems to have fixed my problem, although it has slowed th= ings down a
>>> bit -- presumably because there are 16x more calls to get_rang= e_slices.
>>> While I was in that code I noticed that a new client was being= created for
>>> each batch get. =A0By decreasing the batch size, I've incr= eased this
>>> overhead. =A0I'm thinking of re-writing ColumnFamilyRecord= Reader to do some
>>> connection pooling. =A0Anyone have any thoughts on that?
>>> joost.
>>>
>
>

--0016364ee0ecfabf5b0484e86542--