hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <>
Subject [jira] [Commented] (HIVE-13249) Hard upper bound on number of open transactions
Date Wed, 16 Mar 2016 01:49:33 GMT


Eugene Koifman commented on HIVE-13249:

AcidHouseKeeper uses a ScheduledExecutorService.  It run multiple tasks on separate schedules.
  I don't think there was ever a proposal to run any threads on client side.

The idea Wei and I discussed was to run a single thread per  metastore JVM to count number
of txns periodically and check the computed value in TxnHandler.openTxnx() each time it's
I think this is conceptually the same as your idea.

It's easy enough to have HouseKeeper run multiple tasks, but it complicates testing since
it makes it harder to just run one iteration of a particular task.  We'd need to do some refactoring
in HouseKeepers to make sure this is possible - then they can be combined into a single HouseKeeper
that runs multiple periodic tasks.

Wei, I said earlier that putting this computation in AcidHouseKeeper was a bad idea but I
was wrong.  Since there is a single AcidHouseKeeper per JVM, the task that it runs can easily
just set a static variable on TxnHandler with results of the computation which openTxns()
can read.

As far as testing, look at TestTxnHandler for example, there are multiple examples openTxns()
calls.  In fact each call can open many txnxs at once.  TestTxnCommands.testTimeOutReaper()
has an example on how to run the HouseKeeper in UT, but like I said, you'd need to refactor
it a bit if you want to run multiple tasks in it.

> Hard upper bound on number of open transactions
> -----------------------------------------------
>                 Key: HIVE-13249
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>    Affects Versions: 2.0.0
>            Reporter: Wei Zheng
>            Assignee: Wei Zheng
>         Attachments: HIVE-13249.1.patch
> We need to have a safeguard by adding an upper bound for open transactions to avoid huge
number of open-transaction requests, usually due to improper configuration of clients such
as Storm.
> Once that limit is reached, clients will start failing.

This message was sent by Atlassian JIRA

View raw message