hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-16361) Automatically kill runaway processes
Date Mon, 03 Apr 2017 20:36:41 GMT

     [ https://issues.apache.org/jira/browse/HIVE-16361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eugene Koifman updated HIVE-16361:
----------------------------------
    Description: 
HIVE-13249 added an enforceable limit on how many transactions can be opened concurrently
where the system starts to reject new work to prevent the system getting to a point where
it cannot manage the load.

Another condition to guard against is a runaway process (which would usually be some app (e.g.
Storm) using Streaming Ingest API) that create a very large number of transactions very quickly
all of which immediately get aborted due to some misconfiguration.  This can cause large amount
of metatdata to accumulate in the ACID system slowing everything down and causing instability.


Now that we have TXNS.TXN_AGENT_INFO information we could probably use that refuse work from
a client even before we open any txns if it passes some "runaway client" heuristic.

This is like an unintentional DOS attack

  was:
HIVE-13249 added an enforceable limit on how many transactions can be opened concurrently
where the system starts to reject new work to prevent the system getting to a point where
it cannot manage the load.

Another condition to guard against is a runaway process (which would usually be some app (e.g.
Storm) using Streaming Ingest API) that create a very large number of transactions very quickly
all of which immediately get aborted due to some misconfiguration.  This can cause large amount
of metatdata to accumulate in the ACID system slowing everything down and causing instability.


Now that we have TXNS.TXN_AGENT_INFO information we could probably use that refuse work from
a client even before we open any txns if it passes some "runaway client" heuristic.

This is like DOS attack


> Automatically kill runaway processes 
> -------------------------------------
>
>                 Key: HIVE-16361
>                 URL: https://issues.apache.org/jira/browse/HIVE-16361
>             Project: Hive
>          Issue Type: Improvement
>          Components: Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Priority: Critical
>
> HIVE-13249 added an enforceable limit on how many transactions can be opened concurrently
where the system starts to reject new work to prevent the system getting to a point where
it cannot manage the load.
> Another condition to guard against is a runaway process (which would usually be some
app (e.g. Storm) using Streaming Ingest API) that create a very large number of transactions
very quickly all of which immediately get aborted due to some misconfiguration.  This can
cause large amount of metatdata to accumulate in the ACID system slowing everything down and
causing instability.
> Now that we have TXNS.TXN_AGENT_INFO information we could probably use that refuse work
from a client even before we open any txns if it passes some "runaway client" heuristic.
> This is like an unintentional DOS attack



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message