Return-Path: 
 <issues-return-52355-apmail-hbase-issues-archive=hbase.apache.org@hbase.apache.org>
X-Original-To: apmail-hbase-issues-archive@www.apache.org
Delivered-To: apmail-hbase-issues-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 85F189E8E
	for <apmail-hbase-issues-archive@www.apache.org>;
 Thu, 21 Jun 2012 18:12:43 +0000 (UTC)
Received: (qmail 4743 invoked by uid 500); 21 Jun 2012 18:12:43 -0000
Delivered-To: apmail-hbase-issues-archive@hbase.apache.org
Received: (qmail 4524 invoked by uid 500); 21 Jun 2012 18:12:43 -0000
Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:issues-help@hbase.apache.org>
List-Unsubscribe: <mailto:issues-unsubscribe@hbase.apache.org>
List-Post: <mailto:issues@hbase.apache.org>
List-Id: <issues.hbase.apache.org>
Delivered-To: mailing list issues@hbase.apache.org
Received: (qmail 4409 invoked by uid 99); 21 Jun 2012 18:12:43 -0000
Received: from issues-vm.apache.org (HELO issues-vm) (140.211.11.160)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Jun 2012 18:12:43 +0000
Received: from isssues-vm.apache.org (localhost [127.0.0.1])
	by issues-vm (Postfix) with ESMTP id D88241427F2
	for <issues@hbase.apache.org>; Thu, 21 Jun 2012 18:12:42 +0000 (UTC)
Date: Thu, 21 Jun 2012 18:12:42 +0000 (UTC)
From: "Ted Tuttle (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <1179811209.40168.1340302362888.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Created] (HBASE-6254) deletes w/ many column qualifiers
 overwhelm Region Server
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Ted Tuttle created HBASE-6254:
---------------------------------

             Summary: deletes w/ many column qualifiers overwhelm Region Server
                 Key: HBASE-6254
                 URL: https://issues.apache.org/jira/browse/HBASE-6254
             Project: HBase
          Issue Type: Bug
          Components: performance, regionserver
    Affects Versions: 0.94.0
         Environment: 5 node Cent OS + 1 master, v0.94 on cdh3u3
            Reporter: Ted Tuttle


Execution of Deletes constructed with thousands of calls to Delete.deleteColumn(family, qualifier) are very expensive and slow.

On our (quiet) cluster, a Delete w/ 20k qualifiers took about 13s to complete (as measured by client).

When 10 such Deletes were sent to the cluster via HTable.delete(List<Delete>), one of RegionServers ended up w/ 5 of the requests and became 100% CPU utilized for about 1 hour.

This lead to the client timing out after 20min (2min x 10 retries).  In one case, the client was able to fill the RPC callqueue and received the following error:

  Failed all from region=<region>,hostname=<host>, port=<port> java.util.concurrent.ExecutionException: java.io.IOException: Call queue is full, is ipc.server.max.callqueue.size too small?

Based on feedback (http://search-hadoop.com/m/yITsc1WcDWP), I switched to Delete.deleteColumn(family, qual, timestamp) where timestamp came from KeyValue retrieved from scan based on domain objects.  This version of the delete ran in about 500ms.

User group thread titled "RS unresponsive after series of deletes" has related logs and stacktraces.  

Link to thread: http://search-hadoop.com/m/RmIyr1WcDWP

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira