Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E6F9DB2F for ; Mon, 25 Jun 2012 16:12:17 +0000 (UTC) Received: (qmail 26470 invoked by uid 500); 25 Jun 2012 16:12:16 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 26420 invoked by uid 500); 25 Jun 2012 16:12:16 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 26412 invoked by uid 99); 25 Jun 2012 16:12:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jun 2012 16:12:16 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 74.125.82.169 as permitted sender) Received: from [74.125.82.169] (HELO mail-we0-f169.google.com) (74.125.82.169) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Jun 2012 16:12:11 +0000 Received: by wefh52 with SMTP id h52so3933163wef.14 for ; Mon, 25 Jun 2012 09:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=bJk5RtDdh4lqKLKuOOzAFYL49nQrK9OdMpebJ9IvbcU=; b=msXzs+Dzh3KrGWK5dImUxm1EyUAbV+QA1xXdDbOhxEsLVA+R5lYVLFk9XAeLysMVu8 JCAw8WYN5zi8EqTNQerFG9UOCegK9o0rPXATyo9ia5CcvcSOKE//vxFlIvX750qMou6v R6tNItFasFboir5wa4Lgfc9uNGxkwmnHn0AJ5y7HONcD7+Bus/jQtNjCIwQlK+920OgA cV1d/u3UuOd0tyldJA4LdhHwsiPeC3N1eDNj3LqJUHAnzPxqckfY0gu8xJ1PqFx+DKkq iAlUJBLA9wyA6rO8IaWgAciWVKSy2Cys98xmxSVZasBfKyhjB43vvDqKRnGYBhrNAs7+ BY9g== MIME-Version: 1.0 Received: by 10.180.86.194 with SMTP id r2mr25665083wiz.15.1340640710147; Mon, 25 Jun 2012 09:11:50 -0700 (PDT) Received: by 10.216.12.205 with HTTP; Mon, 25 Jun 2012 09:11:50 -0700 (PDT) In-Reply-To: References: <0CE69E9126D0344088798A3B7F7F808631134337@szxeml531-mbx.china.huawei.com> <7DD701E1-C664-4905-8B29-00618F8C2DB4@gmail.com> <0CE69E9126D0344088798A3B7F7F808631134BA5@szxeml531-mbx.china.huawei.com> Date: Mon, 25 Jun 2012 09:11:50 -0700 Message-ID: Subject: Re: Can there be a doMiniBatchDelete in HRegion? From: Ted Yu To: dev@hbase.apache.org Content-Type: multipart/alternative; boundary=f46d044286dc4cbdb604c34e3d49 X-Virus-Checked: Checked by ClamAV on apache.org --f46d044286dc4cbdb604c34e3d49 Content-Type: text/plain; charset=ISO-8859-1 >From another thread, the following is related to the optimization Anoop is testing: In HRegionServer: public MultiResponse multi(MultiAction multi) throws IOException { ... for (Action a : actionsForRegion) { action = a.getAction(); ... if (action instanceof Delete) { delete(regionName, (Delete) action); I think if we group the deletes of actionsForRegion, we can utilize the following: public int delete(final byte[] regionName, final List deletes) On Mon, Jun 25, 2012 at 6:56 AM, Ted Yu wrote: > After testing in the cluster, please open a JIRA and attach result there. > > Thanks for your effort, Anoop. > > > On Mon, Jun 25, 2012 at 6:50 AM, Anoop Sam John wrote: > >> I have made the new miniBatchDelete () and made the >> HTable#delete(List) to call this new batch delete. >> Just tested initially with the one node cluster. In that itself I am >> getting a performance boost which is very much promising. >> Only one CF and qualifier. >> 10K total rows delete with a batch of 100 deletes. Only deletes happening >> on the table from one thread. >> With the new way the net time taken is reduced by more than 1/10 >> Will test in a 4 node cluster also. I think it will worth doing this >> change. >> >> -Anoop- >> ________________________________________ >> From: yuzhihong@gmail.com [yuzhihong@gmail.com] >> Sent: Wednesday, June 20, 2012 6:31 PM >> To: dev@hbase.apache.org >> Cc: dev@hbase.apache.org >> Subject: Re: Can there be a doMiniBatchDelete in HRegion? >> >> I think you can issue large number of deletes on the same region and >> observe whether the proposed new method gives us performance boost. >> >> Thanks >> >> >> >> On Jun 20, 2012, at 2:49 AM, Anoop Sam John wrote: >> >> > Hi Devs >> > >> > There is a batch put support in the HRegion level. When >> the put(List) happens from client, Puts corresponding to one region >> might get grouped together and handled as a batch.[Depending on the >> availability of rowlocks.. code in HRegion#doMiniBatchPut] For this batch >> there will be single write and sync into the HLog file. >> > >> > >> > >> > A similar kind of delete operation, I am not able to see in HRegion. >> The HTable#delete(List) groups the Deletes for the same RS and make >> one n/w call only. But within the RS, there will be N number of delete >> calls on the region one by one. This will include N number of HLog write >> and sync. If this also can be grouped can we get better performance for the >> multi row delete. Is there any problem in doing this batch delete? I am >> not sure any JIRA is already present for this. >> > >> > >> > >> > Note : Hregion#mutateRowsWithLock().. we do batch operations of Puts >> and Deletes(also) >> > >> > >> > >> > -Anoop- >> > > --f46d044286dc4cbdb604c34e3d49--