hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brock Noland (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7926) long-lived daemons for query fragment execution, I/O and caching
Date Sat, 06 Sep 2014 23:24:28 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124708#comment-14124708
] 

Brock Noland commented on HIVE-7926:
------------------------------------

bq. Small/short queries can be largely processed by this daemon directly, while any heavy
lifting will be performed in standard YARN containers.

Can you speak to how this would work? For example, how would a request flow through the system
in both cases with LLAP running? How would the decision be made to send a request "directly"
to LLAP?



> long-lived daemons for query fragment execution, I/O and caching
> ----------------------------------------------------------------
>
>                 Key: HIVE-7926
>                 URL: https://issues.apache.org/jira/browse/HIVE-7926
>             Project: Hive
>          Issue Type: New Feature
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: LLAPdesigndocument.pdf
>
>
> We are proposing a new execution model for Hive that is a combination of existing process-based
tasks and long-lived daemons running on worker nodes. These nodes can take care of efficient
I/O, caching and query fragment execution, while heavy lifting like most joins, ordering,
etc. can be handled by tasks.
> The proposed model is not a 2-system solution for small and large queries; neither it
is a separate execution engine like MR or Tez. It can be used by any Hive execution engine,
if support is added; in future even external products (e.g. Pig) can use it.
> The document with high-level design we are proposing will be attached shortly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message