hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <>
Subject [jira] [Resolved] (HIVE-7926) long-lived daemons for query fragment execution, I/O and caching
Date Wed, 18 Nov 2015 20:26:11 GMT


Sergey Shelukhin resolved HIVE-7926.
       Resolution: Fixed
    Fix Version/s: 2.0.0
     Release Note: 
LLAP is the new hybrid execution model that enables efficiencies across queries, such as caching
of columnar data, JIT-friendly operator pipelines, and reduced overhead for multiple queries
(including concurrent queries), as well as new performance features like asynchronous I/O,
pre-fetching and multi-threaded processing. The hybrid model consists of a long-lived service
interacting with on-demand elastic containers serving as a tightly integrated DAG-based framework
for query execution. 

The first version of LLAP is being shipped in Hive 2.0 release. The component has been extensively
exercised on test and live clusters, and tested, but is expected to have rough edges in this
initial release.
The current limitations are: supported with Tez only; does not support ACID tables; the I/O
elevator and cache only support ORC format and vectorized execution.

Resolving this JIRA. Without bulk subtask move I cannot move the unfinished subtasks out to
tasks (until we have Jira 7), so they will stay as subtasks for now.

The core functionality will be shipped in 2.0 release. 

> long-lived daemons for query fragment execution, I/O and caching
> ----------------------------------------------------------------
>                 Key: HIVE-7926
>                 URL:
>             Project: Hive
>          Issue Type: New Feature
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>             Fix For: 2.0.0
>         Attachments: LLAPdesigndocument.pdf
> We are proposing a new execution model for Hive that is a combination of existing process-based
tasks and long-lived daemons running on worker nodes. These nodes can take care of efficient
I/O, caching and query fragment execution, while heavy lifting like most joins, ordering,
etc. can be handled by tasks.
> The proposed model is not a 2-system solution for small and large queries; neither it
is a separate execution engine like MR or Tez. It can be used by any Hive execution engine,
if support is added; in future even external products (e.g. Pig) can use it.
> The document with high-level design we are proposing will be attached shortly.

This message was sent by Atlassian JIRA

View raw message