Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2207C10598 for ; Wed, 20 Nov 2013 08:39:26 +0000 (UTC) Received: (qmail 87219 invoked by uid 500); 20 Nov 2013 08:39:24 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 87067 invoked by uid 500); 20 Nov 2013 08:39:23 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 86858 invoked by uid 500); 20 Nov 2013 08:39:22 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 86848 invoked by uid 99); 20 Nov 2013 08:39:22 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Nov 2013 08:39:22 +0000 Date: Wed, 20 Nov 2013 08:39:22 +0000 (UTC) From: "Jason Dere (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-4518) Counter Strike: Operation Operator MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13827439#comment-13827439 ] Jason Dere commented on HIVE-4518: ---------------------------------- Do you mean we can just use fixed names rather than allowing configurable values for those counter names? You're probably right, no need for those names to be configurable, as long as we can configure the counter group for these counters. Let me know if that is what you mean, I will make the change if so. > Counter Strike: Operation Operator > ---------------------------------- > > Key: HIVE-4518 > URL: https://issues.apache.org/jira/browse/HIVE-4518 > Project: Hive > Issue Type: Improvement > Reporter: Gunther Hagleitner > Assignee: Gunther Hagleitner > Attachments: HIVE-4518.1.patch, HIVE-4518.10.patch, HIVE-4518.2.patch, HIVE-4518.3.patch, HIVE-4518.4.patch, HIVE-4518.5.patch, HIVE-4518.6.patch.txt, HIVE-4518.7.patch, HIVE-4518.8.patch, HIVE-4518.9.patch > > > Queries of the form: > from foo > insert overwrite table bar partition (p) select ... > insert overwrite table bar partition (p) select ... > insert overwrite table bar partition (p) select ... > Generate a huge amount of counters. The reason is that task.progress is turned on for dynamic partitioning queries. > The counters not only make queries slower than necessary (up to 50%) you will also eventually run out. That's because we're wrapping them in enum values to comply with hadoop 0.17. > The real reason we turn task.progress on is that we need CREATED_FILES and FATAL counters to ensure dynamic partitioning queries don't go haywire. > The counters have counter-intuitive names like C1 through C1000 and don't seem really useful by themselves. > With hadoop 20+ you don't need to wrap the counters anymore, each operator can simply create and increment counters. That should simplify the code a lot. -- This message was sent by Atlassian JIRA (v6.1#6144)