Return-Path: X-Original-To: apmail-hive-issues-archive@minotaur.apache.org Delivered-To: apmail-hive-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A28CA19594 for ; Wed, 16 Mar 2016 01:49:33 +0000 (UTC) Received: (qmail 54915 invoked by uid 500); 16 Mar 2016 01:49:33 -0000 Delivered-To: apmail-hive-issues-archive@hive.apache.org Received: (qmail 54891 invoked by uid 500); 16 Mar 2016 01:49:33 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 54877 invoked by uid 99); 16 Mar 2016 01:49:33 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Mar 2016 01:49:33 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 741A32C1F58 for ; Wed, 16 Mar 2016 01:49:33 +0000 (UTC) Date: Wed, 16 Mar 2016 01:49:33 +0000 (UTC) From: "Eugene Koifman (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-13249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15196611#comment-15196611 ] Eugene Koifman commented on HIVE-13249: --------------------------------------- AcidHouseKeeper uses a ScheduledExecutorService. It run multiple tasks on separate schedules. I don't think there was ever a proposal to run any threads on client side. The idea Wei and I discussed was to run a single thread per metastore JVM to count number of txns periodically and check the computed value in TxnHandler.openTxnx() each time it's called. I think this is conceptually the same as your idea. It's easy enough to have HouseKeeper run multiple tasks, but it complicates testing since it makes it harder to just run one iteration of a particular task. We'd need to do some refactoring in HouseKeepers to make sure this is possible - then they can be combined into a single HouseKeeper that runs multiple periodic tasks. Wei, I said earlier that putting this computation in AcidHouseKeeper was a bad idea but I was wrong. Since there is a single AcidHouseKeeper per JVM, the task that it runs can easily just set a static variable on TxnHandler with results of the computation which openTxns() can read. As far as testing, look at TestTxnHandler for example, there are multiple examples openTxns() calls. In fact each call can open many txnxs at once. TestTxnCommands.testTimeOutReaper() has an example on how to run the HouseKeeper in UT, but like I said, you'd need to refactor it a bit if you want to run multiple tasks in it. > Hard upper bound on number of open transactions > ----------------------------------------------- > > Key: HIVE-13249 > URL: https://issues.apache.org/jira/browse/HIVE-13249 > Project: Hive > Issue Type: Improvement > Components: Transactions > Affects Versions: 2.0.0 > Reporter: Wei Zheng > Assignee: Wei Zheng > Attachments: HIVE-13249.1.patch > > > We need to have a safeguard by adding an upper bound for open transactions to avoid huge number of open-transaction requests, usually due to improper configuration of clients such as Storm. > Once that limit is reached, clients will start failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)