Return-Path: 
 <dev-return-23517-apmail-hbase-dev-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-dev-archive@www.apache.org
Delivered-To: apmail-hbase-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id ABD6C6A9F
	for <apmail-hbase-dev-archive@www.apache.org>;
 Wed, 27 Jul 2011 12:13:08 +0000 (UTC)
Received: (qmail 92402 invoked by uid 500); 27 Jul 2011 12:13:07 -0000
Delivered-To: apmail-hbase-dev-archive@hbase.apache.org
Received: (qmail 92213 invoked by uid 500); 27 Jul 2011 12:13:04 -0000
Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@hbase.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@hbase.apache.org>
List-Post: <mailto:dev@hbase.apache.org>
List-Id: <dev.hbase.apache.org>
Reply-To: dev@hbase.apache.org
Delivered-To: mailing list dev@hbase.apache.org
Received: (qmail 92198 invoked by uid 99); 27 Jul 2011 12:13:02 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jul 2011 12:13:02 +0000
X-ASF-Spam-Status: No, hits=0.7 required=5.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [69.18.222.47] (HELO smtp1.4emm.com) (69.18.222.47)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jul 2011 12:12:56 +0000
Received: from EX2K7VS03.4emm.local ([192.168.160.203]) by HUB01.4emm.local
 ([192.168.161.132]) with mapi; Wed, 27 Jul 2011 08:12:35 -0400
From: Doug Meil <doug.meil@explorysmedical.com>
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
CC: "apurtell@apache.org" <apurtell@apache.org>
Date: Wed, 27 Jul 2011 08:08:48 -0400
Subject: Re: HBASE-4142?
Thread-Topic: HBASE-4142?
Thread-Index: AcxMVfD6RKnD6dazQiyBrV1VPVKkbw==
Message-ID: <CA557953.ADA9%doug.meil@explorysmedical.com>
In-Reply-To: <CA557605.ADA5%doug.meil@explorysmedical.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-MacOutlook/14.12.0.110505
acceptlanguage: en-US
Content-Type: text/plain; charset="iso-8859-2"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org


Created HBASE-4143



On 7/27/11 7:54 AM, "Doug Meil" <doug.meil@explorysmedical.com> wrote:

>
>I think that would be a good idea.
>
>
>
>On 7/26/11 10:52 PM, "Ted Yu" <yuzhihong@gmail.com> wrote:
>
>>Should we check writebuffer size in the for loop, say after every 5 puts
>>are added ?
>>
>>On Jul 26, 2011, at 7:43 PM, Doug Meil <doug.meil@explorysmedical.com>
>>wrote:
>>
>>>=20
>>> But as long as autoFlush=3Dfalse, both put() methods use the writebuffe=
r.
>>> Was the issue that the flush evaluation doesn't happen on
>>>put(List<Put>)
>>> until the entire list was processed?
>>>=20
>>> If that was the issue, then I think it would make sense to call that
>>>out
>>> explicitly in the Javadoc rather than just saying it may cause perf
>>> problems.
>>>=20
>>>=20
>>>=20
>>> public void put(final Put put) throws IOException {
>>>    doPut(Arrays.asList(put));
>>>  }
>>>=20
>>>  /**
>>>   * {@inheritDoc}
>>>   */
>>>  @Override
>>>  public void put(final List<Put> puts) throws IOException {
>>>    doPut(puts);
>>>  }
>>>=20
>>>  private void doPut(final List<Put> puts) throws IOException {
>>>    for (Put put : puts) {
>>>      validatePut(put);
>>>      writeBuffer.add(put);
>>>      currentWriteBufferSize +=3D put.heapSize();
>>>    }
>>>    if (autoFlush || currentWriteBufferSize > writeBufferSize) {
>>>      flushCommits();
>>>    }
>>>  }
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>> On 7/26/11 10:28 PM, "Andrew Purtell" <apurtell@apache.org> wrote:
>>>=20
>>>> We were confronted with a MapReduce job, developed by an internal dev
>>>> group, with reducers writing to HBase directly that would see a
>>>>fraction
>>>> of the reducers consistently killed due to timeout. Looking at jstacks
>>>> all seemed well -- RPC in progress, nothing remarkable client or
>>>>server
>>>> side. We were stumped until we looked at the client code. It turns out
>>>> they would, in a data driven way, occasionally submit really really
>>>> really large lists. After we asked them to use the write buffer, all
>>>>was
>>>> well.
>>>>=20
>>>> Best regards,
>>>>=20
>>>>=20
>>>>   - Andy
>>>>=20
>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>>Hein
>>>> (via Tom White)
>>>>=20
>>>>=20
>>>>> ________________________________
>>>>> From: Doug Meil <doug.meil@explorysmedical.com>
>>>>> To: "dev@hbase.apache.org" <dev@hbase.apache.org>
>>>>> Sent: Tuesday, July 26, 2011 7:14 PM
>>>>> Subject: HBASE-4142?
>>>>>=20
>>>>> Hi there-
>>>>>=20
>>>>> I just saw this in the build message=A9
>>>>>=20
>>>>> HBASE-4142 Advise against large batches in javadoc for
>>>>> HTable#put(List<Put>)
>>>>>=20
>>>>> =A9 And I was curious as to why this was a bad thing.  We do this and
>>>>>it
>>>>> actually is quite helpful (in concert with an internal utility class
>>>>> that later became HTableUtil).
>>>>>=20
>>>>>=20
>>>>>=20
>>>>> Doug Meil
>>>>> Chief Software Architect, Explorys
>>>>> doug.meil@explorys.com
>>>>>=20
>>>>>=20
>>>>>=20
>>>=20
>